Image processing method, device, apparatus and storage medium

Abstract

Embodiments of the present application provide an image processing method, device and equipment, and a storage medium, belonging to the field of image processing. The method comprises: acquiring a plurality of first images and acquiring a target motion mask, the target motion mask being used to describe whether an image region where each image unit in each first image is located is a motion region or a static region; performing HDR fusion processing on the plurality of first images according to the target motion mask to obtain a second image, wherein for each image unit, in a case where it is determined according to the target motion mask that the image region where the image unit is located is a motion region, performing HDR fusion processing based on pixel values of the image unit in a first number of first images, and in a case where it is determined according to the target motion mask that the image region where the image unit is located is a static region, performing HDR fusion processing based on detail information of the image unit in a second number of first images, the first number being less than the second number. The technical solution of the embodiments of the present application improves the quality of the image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Currently, an increasing number of head-mounted devices integrate cameras. For example, smart glasses often incorporate cameras on their frames or temples to capture real-time images or videos from a first-person perspective. However, some smart glasses, such as augmented reality (AR) glasses, require the integration of numerous components. Limited by their optical structure and size design, their lenses have lower light intake. Furthermore, due to the motion they generate during use, the signal-to-noise ratio of single-frame images acquired under normal lighting conditions is low. For instance, while maintaining image brightness, areas with human figures or other moving targets are prone to noticeable noise accumulation, blurred details, or enhanced blocky noise, especially in moving areas, thus affecting the overall image clarity and usability.

[0003] In related technologies, multi-frame High Dynamic Range (HDR) fusion methods are commonly used to optimize all pixel regions of the entire image using a unified strategy. However, this approach is mainly suitable for general scenarios or specific electronic devices, and it is prone to problems when used on smart glasses. For example, when there are moving targets in the scene, the registration error will be directly passed to the fusion weight calculation, making it impossible for the pixels of the moving region in the image to correspond accurately between frames. In this case, the unified strategy is difficult to correctly estimate the effective information contribution of each frame, resulting in noise accumulation or motion artifacts in the fused moving region, leading to poor quality of the final output image. Summary of the Invention

[0004] This invention provides an image processing method, apparatus, device, and storage medium, aiming to improve image quality in head-mounted device usage scenarios.

[0005] In a first aspect, an image processing method provided by an embodiment of the present invention includes: Acquire multiple first images and acquire a target motion mask, wherein the target motion mask is used to describe whether the image region where each image unit in each first image is located is a moving region or a static region, and the image unit includes one or more pixels; According to the target motion mask, the plurality of first images are subjected to HDR fusion processing to obtain a second image. The HDR fusion processing of the plurality of first images includes: for each image unit, if the image region where the image unit is located is determined to be a moving region according to the target motion mask, HDR fusion processing is performed based on the pixel values ​​of the image unit in a first number of first images; if the image region where the image unit is located is determined to be a static region according to the target motion mask, HDR fusion processing is performed based on the detail information of the image unit in a second number of first images, where the first number is less than the second number.

[0006] In a second aspect, an image processing apparatus provided by an embodiment of the present invention includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the image processing method as described in the first aspect.

[0007] Thirdly, an embodiment of the present invention provides a head-mounted device including a shooting device and an inertial measurement unit, and further including an image processing device as described in the second aspect.

[0008] Fourthly, embodiments of the present invention provide a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the image processing method as described in the first aspect.

[0009] After implementing the scheme of this invention, by using a target motion mask to describe the motion degree of each image unit in each first image, it can be determined whether the image region where the image unit is located is a moving region or a static region. For the image region where the image unit is located is a moving region, HDR fusion processing is performed based on the pixel values ​​of the image unit in a small number of first images to narrow the time sampling window, thereby avoiding the occurrence of ghosting caused by the misalignment and superposition of moving pixels at different positions, and ensuring the clarity of the moving region. For the image region where the image unit is located is a static region, HDR fusion processing is performed based on the detail information of the image unit in a large number of first images to fully expose and stack, which both widens the dynamic range and cancels temporal noise. This dynamic and static separation processing mechanism solves the physical limitation of "preserving clarity at the expense of dynamic range" or "preserving dynamic range at the expense of ghosting" caused by a globally uniform frame rate, so that the final output image takes into account both clarity and richness in the global visual perception, thus improving the image quality. Attached Figure Description

[0010] Figure 1This is a schematic block diagram of a head-mounted device provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of the present invention; Figure 3 yes Figure 2 A flowchart illustrating a sub-step of the image processing method in the image processing diagram; Figure 4 yes Figure 2 A flowchart illustrating another sub-step of the image processing method in the diagram; Figure 5 yes Figure 2 A flowchart illustrating another sub-step of the image processing method in the diagram; Figure 6 yes Figure 2 A flowchart illustrating another sub-step of the image processing method in the diagram; Figure 7 This is a schematic block diagram of an image processing device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] Currently, an increasing number of head-mounted devices integrate cameras. For example, smart glasses often incorporate cameras on their frames or temples to capture real-time images or videos from a first-person perspective. However, some smart glasses, such as augmented reality (AR) glasses, require the integration of numerous components. Limited by their optical structure and size design, their lenses have lower light intake. Furthermore, due to the motion they generate during use, the signal-to-noise ratio of single-frame images acquired under normal lighting conditions is low. For instance, while maintaining image brightness, areas with human figures or other moving targets are prone to noticeable noise accumulation, blurred details, or enhanced blocky noise, especially in moving areas, thus affecting the overall image clarity and usability.

[0014] In related technologies, multi-frame High Dynamic Range (HDR) fusion methods are commonly used to optimize all pixel regions of the entire image using a unified strategy. However, this approach is mainly suitable for general scenarios or specific electronic devices, and it is prone to problems when used on smart glasses. For example, when there are moving targets in the scene, the registration error will be directly passed to the fusion weight calculation, making it impossible for the pixels of the moving region in the image to correspond accurately between frames. In this case, the unified strategy is difficult to correctly estimate the effective information contribution of each frame, resulting in noise accumulation or motion artifacts in the fused moving region, leading to poor quality of the final output image.

[0015] To address the aforementioned problems, embodiments of the present invention provide an image processing method, an image processing apparatus, a head-mounted device, and a storage medium. Considering that the head-mounted device may include an image processing apparatus, and the image processing apparatus corresponds to a functional implementation unit of the image processing method, for ease of understanding and explanation, the image processing method, image processing apparatus, and head-mounted device provided in the embodiments of this application will be described intermittently below as needed, with other aspects further supplemented and explained on this basis.

[0016] This invention, through a target motion mask describing the motion level of each image unit in each first image, can determine whether the image region where the image unit is located is a moving region or a static region. For image regions where the image unit is located are moving regions, HDR fusion processing is performed based on the pixel values ​​of the image unit in a smaller number of first images to narrow the time sampling window. This avoids ghosting caused by misaligned overlapping of moving pixels at different positions, ensuring the clarity of moving regions. Conversely, for image regions where the image unit is located are static regions, HDR fusion processing is performed based on the detailed information of the image unit in a larger number of first images to ensure sufficient exposure stacking. This widens the dynamic range and cancels out temporal noise. This motion-static separation processing mechanism solves the physical limitations of "sacrificing dynamic range for clarity" or "ghosting for dynamic range" caused by a globally uniform frame rate. This allows the final output image to simultaneously maintain clarity and richness in the overall visual perception, improving image quality.

[0017] In this embodiment of the invention, the image processing method can be applied to head-mounted devices and controllers within the head-mounted devices. The head-mounted devices include smart audio glasses or smart camera glasses without a display device, as well as AR glasses with a display device, etc.

[0018] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings.

[0019] Please see Figure 1 , Figure 1This is a schematic block diagram of a head-mounted device provided in an embodiment of the present invention.

[0020] like Figure 1 As shown, the head-mounted device 100 includes a camera 110, an inertial measurement unit (IMU) 120, and an image processing device 130. The camera 110 is used to acquire images, the inertial measurement unit 120 is used to acquire IMU measurement data synchronized with the images, and the image processing device 130 is communicatively connected to both the camera 110 and the inertial measurement unit 120. The camera 110 may include a camera.

[0021] In some embodiments, the image processing apparatus 130 is configured to perform the following steps: acquiring a plurality of first images and acquiring a target motion mask, the target motion mask being used to describe whether the image region where each image unit in each first image is located is a moving region or a static region, the image unit comprising one or more pixels; performing HDR fusion processing on the plurality of first images according to the target motion mask to obtain a second image, wherein performing HDR fusion processing on the plurality of first images includes: for each image unit, if the image region where the image unit is located is determined to be a moving region according to the target motion mask, performing HDR fusion processing based on the pixel values ​​of the image unit in a first number of first images; if the image region where the image unit is located is determined to be a static region according to the target motion mask, performing HDR fusion processing based on the detail information of the image unit in a second number of first images, the first number being less than the second number.

[0022] In some embodiments, when acquiring a plurality of first images, the image processing device 130 is configured to: acquire an image frame sequence captured by the imaging device of a head-mounted device and IMU measurement data synchronized with the image frame sequence acquired by the inertial measurement unit of the head-mounted device; determine an inter-frame rotation matrix of each image frame in the image frame sequence relative to a reference frame in the image frame sequence based on the IMU measurement data; determine a homography transformation matrix corresponding to each image frame based on the inter-frame rotation matrix of each image frame in the image frame sequence relative to the reference frame, the extrinsic rotation matrix between the inertial measurement unit and the imaging device, and the intrinsic parameter matrix of the imaging device; and apply the corresponding homography transformation matrix to each image frame to perform perspective transformation, thereby obtaining the plurality of first images spatially aligned with the reference frame.

[0023] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the head-mounted device described above can be referred to the corresponding process in the following image processing method embodiments, and will not be repeated here.

[0024] The following will combine Figure 1 The head-mounted device described in the text provides a detailed explanation of the image processing method provided in the embodiments of the present invention. It should be noted that... Figure 1 The head-mounted device described herein is only used to explain the image processing method provided in the embodiments of the present invention, but does not constitute a limitation on the application scenarios of the image processing method provided in the embodiments of the present invention.

[0025] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of the present invention.

[0026] like Figure 2 As shown, the image processing method includes steps S101 to S102.

[0027] Step S101: Acquire multiple first images and acquire a target motion mask. The target motion mask is used to describe whether the image region where each image unit in each first image is located is a moving region or a static region. The image unit includes one or more pixels.

[0028] In this embodiment, the target motion mask M(x,y)∈[0,1] includes the probability that the image region where each image unit is located is a moving region. If the probability that the image region where the image unit is located is a moving region is greater than or equal to a preset probability, the image region where the image unit is located is determined to be a moving region. If the probability that the image region where the image unit is located is a moving region is less than the preset probability, the image region where the image unit is located is determined to be a static region. Among them, the multiple first images are continuous.

[0029] In some embodiments, such as Figure 3 As shown, obtaining multiple first images in step S101 includes sub-steps S1011 to S1014.

[0030] Sub-step S1011: Acquire the image frame sequence captured by the imaging device of the head-mounted device and the IMU measurement data synchronized with the image frame sequence acquired by the inertial measurement unit of the head-mounted device.

[0031] In this embodiment, the image frame sequence includes multiple consecutive raw image frames; for example, the image frame sequence includes six consecutive raw image frames. The raw image frames are output by the imaging device, and the IMU measurement data includes angular velocity data and acceleration data. In some embodiments, the exposure parameters of the multiple raw image frames are different.

[0032] Sub-step S1012: Determine the inter-frame rotation matrix of each image frame in the image frame sequence relative to the reference frame in the image frame sequence based on the IMU measurement data.

[0033] In this embodiment, the reference frame is the image frame with the best image quality in the image frame sequence, or the reference frame is the newest image frame in the image frame sequence.

[0034] In some embodiments, determining the inter-frame rotation matrix of each image frame in the image frame sequence relative to a reference frame in the image frame sequence based on IMU measurement data includes: for each image frame, obtaining attitude change information of the imaging device during the acquisition of the image frame to the reference frame from the IMU measurement data, and determining the inter-frame rotation matrix of the image frame relative to the reference frame based on the attitude change information.

[0035] Sub-step S1013: Determine the homography transformation matrix corresponding to each image frame based on the inter-frame rotation matrix of each image frame relative to the reference frame, the extrinsic rotation matrix between the inertial measurement unit and the imaging device, and the intrinsic parameter matrix of the imaging device.

[0036] In this embodiment, the homography transformation matrix of the i-th image frame in the image frame sequence relative to the reference frame can be expressed as: , K is the intrinsic parameter matrix of the imaging device, R mount R is the extrinsic rotation matrix between the inertial measurement unit and the imaging device. imu,i It is the inter-frame rotation matrix of the i-th image frame in the image frame sequence relative to the reference frame.

[0037] Sub-step S1014: Apply the corresponding homography transformation matrix to each image frame to perform perspective transformation, and obtain multiple first images that are spatially aligned with the reference frame.

[0038] In this embodiment, the inter-frame rotation matrix is ​​derived entirely from IMU measurement data, without relying on image feature extraction and matching. This ensures stable and accurate inter-frame rotation even in visually degraded scenarios such as low texture, low lighting, motion blur, or repetitive textures, effectively avoiding alignment failures or image jitter caused by feature loss. Simultaneously, by combining a precise IMU-camera extrinsic rotation matrix with camera intrinsic parameters, the physical rotation is accurately mapped to the homography transformation of the pixel plane, effectively compensating for pose changes during head-mounted device movement. This eliminates unintended global jitter and provides a stable and geometrically consistent visual foundation for subsequent HDR fusion.

[0039] In some embodiments, such as Figure 4 As shown, obtaining the target motion mask in step S101 includes sub-steps S1015 to S1018.

[0040] Sub-step S1015: Determine the motion intensity parameters of the head-mounted device based on the IMU measurement data.

[0041] In this embodiment, the motion intensity parameter of the head-mounted device is used to describe the degree of motion of the head-mounted device during the acquisition of image frame sequences, including the maximum angular velocity and maximum rotation angle of the head-mounted device.

[0042] Sub-step S1016: Determine the first motion vector of each image unit in the reference image relative to each first image based on the homography transformation matrix of each first image relative to the reference image in the plurality of first images.

[0043] In this embodiment, the position of each image unit in the reference image in the first image can be determined based on the homography transformation matrix of the first image relative to the reference image. Based on the position of the image unit in the reference image and its position in the first image, the first motion vector of the image unit can be determined. For example, if an image unit includes multiple pixels, the motion vector of each pixel in the image unit can be determined based on the position of each pixel in the reference image and the first image. The largest motion vector among the motion vectors of each pixel in the image unit is determined as the first motion vector of the image unit. Alternatively, an average motion vector can be determined based on the motion vectors of each pixel in the image unit, and this average motion vector is then determined as the first motion vector of the image unit.

[0044] Sub-step S1017: Determine the first motion amplitude of each image unit in the reference image relative to each first image based on the first motion vector of each image unit in the reference image relative to each first image.

[0045] In this embodiment, the first motion vector of each image unit in the reference image relative to each first image can be calculated to obtain the first motion amplitude of each image unit in the reference image relative to each first image.

[0046] Sub-step S1018: Generate a target motion mask based on the first motion amplitude and motion intensity parameters of each image unit in the reference image relative to each first image.

[0047] This embodiment introduces motion intensity parameters measured by IMU as a global motion benchmark and coordinates them with the first motion amplitude derived from homography transformation for judgment. This effectively decouples the background parallax caused by the head-mounted device's own motion from the actual displacement of independent targets within the scene. It avoids the technical defect of misjudging device jitter or rotation as moving foreground, improves the segmentation purity and anti-global interference capability of the motion mask in dynamic device usage scenarios, and ensures that the output target motion mask can accurately map the real dynamic area in the first image, providing a more accurate motion layering basis for subsequent HDR fusion.

[0048] In some embodiments, generating a target motion mask based on a first motion amplitude and motion intensity parameter of each image unit in the reference image relative to each first image includes: for each image unit in the reference image, aggregating the first motion amplitude of the image unit relative to each first image, wherein the aggregated motion amplitude of the image unit is the average amplitude or maximum amplitude of the first motion amplitude of the image unit relative to each first image; determining a segmentation threshold weighting coefficient based on the motion intensity parameter, multiplying a preset benchmark segmentation threshold by the segmentation threshold weighting coefficient and then adding the result to the benchmark segmentation threshold to obtain a target segmentation threshold; marking the image regions in the first image where the aggregated motion intensity is greater than or equal to the target segmentation threshold as motion regions and marking the image regions in the first image where the aggregated motion intensity is less than the target segmentation threshold as static regions to obtain a binary mask; and performing morphological closing and Gaussian blur processing on the binary mask to obtain the target motion mask. The segmentation threshold weighting coefficient is positively correlated with the motion intensity parameter.

[0049] In some embodiments, such as Figure 5 As shown, the step S101 of obtaining the target motion mask also includes: Sub-step S1019: Based on a preset optical flow algorithm, determine the second motion vector of each image unit in the reference image relative to each first image, and determine the second motion amplitude of each image unit in the reference image relative to each first image based on the second motion vector of each image unit in the reference image relative to each first image.

[0050] In this embodiment, the preset optical flow algorithm includes a sparse optical flow algorithm or a dense optical flow algorithm. Specifically, determining the second motion vector of each image unit in the reference image relative to each first image based on the preset optical flow algorithm includes: determining the motion vector of each pixel in the reference image relative to each first image based on the preset optical flow algorithm; for each image unit, the largest motion vector within the motion vector of each pixel in the image unit can be determined as the second motion vector of the image unit, or an average motion vector can be determined based on the motion vector of each pixel in the image unit, and this average motion vector can be determined as the second motion vector of the image unit.

[0051] Sub-step S10181: Determine the target motion amplitude of each image unit in the reference image relative to each first image based on the first motion amplitude and the second motion amplitude of each image unit in the reference image relative to each first image.

[0052] In this embodiment, for each image unit in the reference image, the first motion amplitude and the second motion amplitude of the image unit relative to each first image can be weighted and summed to obtain the target motion amplitude of the image unit relative to each first image.

[0053] In some embodiments, determining the target motion amplitude of each image unit in the reference image relative to each first image, based on a first motion amplitude and a second motion amplitude of each image unit in the reference image relative to each first image, includes: for each image unit in the reference image and each first image, subtracting the first motion amplitude of the image unit relative to the first image from the second motion amplitude of the image unit relative to the first image to obtain a motion amplitude difference; determining a first weighting coefficient and a second weighting coefficient based on the motion amplitude difference, wherein the first weighting coefficient is less than the second weighting coefficient; multiplying the first weighting coefficient by the first motion amplitude to obtain a first weighted motion amplitude; multiplying the second weighting coefficient by the second motion amplitude to obtain a second weighted motion amplitude; and adding the first weighted motion amplitude and the second weighted motion amplitude to obtain the target motion amplitude of the image unit relative to the first image. Wherein, the larger the motion amplitude difference, the larger the difference between the first weighting coefficient and the second weighting coefficient.

[0054] Sub-step S10182: Generate a target motion mask based on the target motion amplitude and motion intensity parameters of each image unit in the reference image relative to each first image.

[0055] This embodiment achieves complementary advantages between the macroscopic physical motion field and the microscopic pixel motion field by fusing the first motion amplitude based on IMU and the second motion amplitude based on optical flow. This improves the representation accuracy and robustness of motion perception in complex scenes. On the one hand, global physical constraints can calibrate and suppress abnormal responses generated by optical flow in visually degraded areas. On the other hand, dense optical flow information can finely depict the non-rigid deformation details of the target boundary, making up for the limitation that homography transformation is only applicable to planar or rigid motion models. In this way, the final determined target motion amplitude has both global stability and local sensitivity, improving the clarity, regional integrity, and anti-interference ability of the generated target motion mask at dynamic target boundaries and the changes in illumination / texture. This provides a more accurate basis for motion layering for subsequent HDR fusion.

[0056] In some embodiments, generating a target motion mask based on the target motion amplitude and motion intensity parameters of each image unit in the reference image relative to each first image includes: for each image unit in the reference image, aggregating the target motion amplitude of the image unit relative to each first image, wherein the aggregated motion amplitude is the maximum motion amplitude among the target motion amplitudes of the image unit relative to each first image; determining a segmentation threshold weighting coefficient based on the motion intensity parameter, multiplying a preset benchmark segmentation threshold by the segmentation threshold weighting coefficient and then adding the result to the benchmark segmentation threshold to obtain a target segmentation threshold; marking the image regions in the first image where the aggregated motion intensity is greater than or equal to the target segmentation threshold as motion regions and marking the image regions in the first image where the aggregated motion intensity is less than the target segmentation threshold as static regions to obtain a binary mask; and performing morphological closing and Gaussian blur processing on the binary mask to obtain the target motion mask. The segmentation threshold weighting coefficient is positively correlated with the motion intensity parameter.

[0057] Step S102: Based on the target motion mask, perform HDR fusion processing on multiple first images to obtain a second image. The HDR fusion processing on multiple first images includes: for each image unit, if the image region where the image unit is located is determined to be a moving region based on the target motion mask, perform HDR fusion processing based on the pixel values ​​of the image unit in a first number of first images; if the image region where the image unit is located is determined to be a static region based on the target motion mask, perform HDR fusion processing based on the detail information of the image unit in a second number of first images, where the first number is less than the second number.

[0058] This embodiment uses a target motion mask to describe the motion level of each image unit in each first image. This allows it to determine whether the image region where the image unit is located is a moving region or a static region. For image regions where the image unit is located, HDR fusion processing is performed based on the pixel values ​​of the image unit in a smaller number of first images to narrow the time sampling window. This avoids ghosting caused by misaligned overlapping of moving pixels at different positions, ensuring the clarity of moving regions. For image regions where the image unit is located, HDR fusion processing is performed based on the detail information of the image unit in a larger number of first images to fully expose and stack the images. This widens the dynamic range and cancels out temporal noise. This motion-static separation processing mechanism solves the physical limitations of "sacrificing dynamic range for clarity" or "ghosting for dynamic range" caused by a globally uniform frame rate. This allows the final output image to simultaneously maintain clarity and richness in the overall visual perception, improving image quality.

[0059] For example, if the image region where the image unit is located is determined to be a moving region based on the target motion mask, HDR fusion processing is performed based on the pixel values ​​of the image unit in the first three frames. If the image region where the image unit is located is determined to be a static region based on the target motion mask, HDR fusion processing is performed based on the detail information of the image unit in the first six frames.

[0060] In some embodiments, such as Figure 6 As shown, step S102 includes sub-steps S1021 to S1023.

[0061] Sub-step S1021: Based on the first strategy, perform HDR fusion processing on a first number of first images to obtain a first fused image. The first strategy is used to suppress motion artifacts in the motion region.

[0062] In this embodiment, the first number of first images includes reference images, and the first strategy includes determining the fusion weight of each first image based on the pixel differences and motion consistency between each first image and the reference image, and performing weighted fusion based on the fusion weight of each first image.

[0063] In some embodiments, based on a first strategy, HDR fusion processing is performed on a first number of first images to obtain a first fused image, including: for each of the first images in the first number of first images, determining the pixel value difference of each image unit in the first image relative to a reference image; determining a weighting coefficient for each image unit in the first image based on the pixel value difference and motion consistency of each image unit in the first image relative to the reference image; for each image unit in the reference image, determining a target pixel value for the image unit based on the pixel value of the image unit in each first image; and replacing the current pixel value of each image unit in the reference image with the corresponding target pixel value to obtain the first fused image. The weighting coefficient is negatively correlated with the pixel value difference. This embodiment achieves differentiated filtering and fusion of multi-frame light and color information by using a reference frame as a spatiotemporal anchor point and utilizing the pixel value difference to inversely map the weighting coefficient. In the dynamic region, due to the drastic fluctuations in pixel values ​​between frames of moving objects, the corresponding weighting coefficients are significantly reduced, thus effectively avoiding the mixing of ghosting and afterimages into the fusion result. In the static region, due to the high consistency between multiple frames, the high weight accumulation and superposition effectively enhances the brightness level and detail expression, thus ensuring that the first fused image has clear outline boundaries and ghosting-free visual quality in the dynamic region.

[0064] In some embodiments, determining the weighting coefficient of each image unit in the first image based on the pixel value difference of each image unit in the first image relative to the reference image includes: determining the motion amplitude deviation between a first motion amplitude and a second motion amplitude of the image unit in the first image relative to the reference image; and determining the weighting coefficient of each image unit in the first image based on the pixel value difference and the motion amplitude deviation of each image unit in the first image relative to the reference image. The weighting coefficient is negatively correlated with the pixel value difference and negatively correlated with the motion amplitude deviation. This embodiment achieves dual collaborative verification of the light color gamut and motion domain by using motion amplitude deviation as a constraint term for weighting coefficients. On the one hand, even if pixel value differences due to illumination fluctuations or texture loss cannot truly reflect the reliability of the region, motion amplitude deviation constraints can still suppress the mixing of potential mismatch information by capturing conflicts in geometric calculations, thus enhancing the adaptability to complex scenes. On the other hand, low weights are assigned to regions with large motion amplitude deviations (usually corresponding to the edges of moving targets or occluded interfaces), effectively preventing the erroneous superposition of foreground and background pixels and edge blurring, ensuring that the first fused image has both a clean, ghost-free performance in dynamic regions and maintains clear and sharp contour boundaries.

[0065] Sub-step S1022: Based on the second strategy, perform HDR fusion processing on the second number of first images to obtain a second fused image. The exposure parameters of the second number of first images are different, and the first strategy is different from the second strategy. The second strategy is used to expand the dynamic range of the static area.

[0066] In this embodiment, the second strategy includes multi-scale fusion of a second number of first images based on multi-scale decomposition and image quality metrics to expand the dynamic range of the fused images.

[0067] In some embodiments, based on a second strategy, performing HDR fusion processing on a second number of first images to obtain a second fused image includes: performing multi-scale decomposition on each of the second number of first images to obtain a multi-scale representation corresponding to each first image, wherein the multi-scale representation includes detail information at multiple scale levels; generating a fusion weight map corresponding to each first image based on the image quality metric parameters of each image unit in each first image, wherein the image quality metric parameters are used to describe the confidence level of the image unit in the HDR fusion processing; performing weighted fusion on the multi-scale representation corresponding to each first image at multiple scale levels based on the fusion weight map corresponding to each first image to obtain a target multi-scale representation; and performing multi-scale reconstruction on the target multi-scale representation to obtain the second fused image. This embodiment achieves refined extraction and reorganization of multi-exposure image information through multi-scale decomposition and hierarchical weighted fusion, overcoming the detail loss and tone distortion defects easily caused by traditional pixel-level fusion. This results in the second fused image possessing a wide dynamic range, rich texture details, and a natural visual feel within a static area, providing a high-quality background image for subsequent mask synthesis with the first fused image.

[0068] In some embodiments, image quality metrics include contrast, saturation, and good exposure, wherein contrast describes the richness of local detail in an image unit, saturation describes the vibrancy of color in an image unit, and good exposure describes the degree to which the brightness of an image unit deviates from extreme values ​​(0 or 255). The multi-scale decomposition is a Laplacian pyramid decomposition, the multi-scale representation is a Laplacian pyramid, and the multi-scale reconstruction is a Laplacian pyramid reconstruction.

[0069] Sub-step S1023: Based on the target motion mask, the first fused image and the second fused image are weighted and synthesized to obtain the second image.

[0070] This embodiment effectively resolves the technical contradiction between "motion ghosting suppression" and "high dynamic range preservation" in traditional HDR fusion by employing targeted optimization fusion strategies for dynamic and static regions and using mask-guided synthesis. On the one hand, it focuses on ghosting suppression in dynamic regions to ensure clear outlines of moving objects and eliminate artifacts in the output image; on the other hand, it focuses on dynamic range expansion in static regions to fully inherit the brightness levels and rich details of multiple exposure frames. This results in a second image that combines regional adaptability with global consistency. In complex lighting scenes containing moving targets, it can simultaneously achieve ghosting-free imaging and wide dynamic range performance across the entire image, improving the imaging quality of head-mounted devices in complex lighting scenes with moving targets.

[0071] In some embodiments, after step S102, the method further includes: determining a first imaging region and a second imaging region in the second image, wherein the first imaging region is located within the boundary corresponding to the original image captured by the camera of the head-mounted device, and the second imaging region is located outside the boundary corresponding to the original image captured by the camera of the head-mounted device; determining the field of view center of the reference image of the head-mounted device in the plurality of first images, and determining a cropping reference center within the first imaging region based on the field of view center; and cropping the first imaging region based on the cropping reference center and the target resolution to obtain the target image. The field of view center of the reference image of the head-mounted device in the plurality of first images is determined based on IMU data acquired by the inertial measurement unit of the head-mounted device. This embodiment achieves intelligent spatial cropping of the effective field of view of the synthesized image by anchoring the cropping reference center to the field of view center of the reference image and combining it with target resolution constraints. This solves the problem of invalid edges and field of view offset introduced by the preceding alignment and fusion processing, and avoids the situation where the target of interest is cut out of the frame due to device jitter. This results in the final target image possessing field of view center alignment fidelity, spatial information purity, and resolution adaptability.

[0072] In some embodiments, after determining the target image, the image processing method provided in this embodiment of the invention further includes: if the size of the target image exceeds the field of view of the camera device of the head-mounted device, reducing the size of the target image so that the size of the reduced target image does not exceed the field of view of the camera device of the head-mounted device.

[0073] In some embodiments, the image processing method provided by this invention can be applied to a system consisting of a head-mounted device and a terminal device, wherein the head-mounted device and the terminal device are communicatively connected. The head-mounted device includes an imaging device and an inertial measurement unit (IMU). The imaging device is used to acquire images, and the inertial measurement unit is used to acquire IMU measurement data synchronized with the images. The communication connection between the head-mounted device and the terminal device can be a wireless communication connection or a wired communication connection.

[0074] For example, the head-mounted device is used to acquire an image frame sequence captured by the shooting device and IMU measurement data synchronized with the image frame sequence acquired by the inertial measurement unit of the head-mounted device, and sends the image frame sequence and IMU measurement data to the terminal device; the terminal device is used to generate multiple first images and a target motion mask based on the IMU measurement data and the image frame sequence; the terminal device is used to perform HDR fusion processing on the multiple first images based on the target motion mask to obtain a second image, wherein the HDR fusion processing on the multiple first images includes: for each image unit, if it is determined from the target motion mask that the image area where the image unit is located is a moving area, HDR fusion processing is performed based on the pixel values ​​of the image unit in a first number of first images; if it is determined from the target motion mask that the image area where the image unit is located is a static area, HDR fusion processing is performed based on the detail information of the image unit in a second number of first images, where the first number is less than the second number; the terminal device is used to send the second image to the head-mounted device.

[0075] For example, the head-mounted device is used to acquire an image frame sequence captured by the shooting device and IMU measurement data synchronized with the image frame sequence acquired by the inertial measurement unit of the head-mounted device, and sends the image frame sequence and IMU measurement data to the terminal device; the terminal device is used to generate multiple first images and a target motion mask according to the IMU measurement data and the image frame sequence, and send the multiple first images and the target motion mask to the AR glasses; when the head-mounted device receives multiple first images and the target motion mask, it performs HDR fusion processing on the multiple first images according to the received target motion mask to obtain a second image, wherein the HDR fusion processing on the multiple first images includes: for each image unit, if it is determined according to the target motion mask that the image area where the image unit is located is a moving area, HDR fusion processing is performed based on the pixel values ​​of the image unit in a first number of first images; if it is determined according to the target motion mask that the image area where the image unit is located is a static area, HDR fusion processing is performed based on the detail information of the image unit in a second number of first images, where the first number is less than the second number.

[0076] Please see Figure 7 , Figure 7 This is a schematic block diagram of the structure of an image processing device provided in an embodiment of the present invention.

[0077] like Figure 7 As shown, the image processing device 130 includes a processor 131 and a memory 132, which are connected via a bus 133, such as an I2C (Inter-integrated Circuit) bus.

[0078] Specifically, processor 131 provides computing and control capabilities to support the operation of the entire image processing device. Processor 131 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0079] Specifically, the memory 132 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0080] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the embodiments of the present invention, and do not constitute a limitation on the image processing apparatus to which the embodiments of the present invention are applied. Specific image processing apparatuses may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0081] The processor 131 is used to run a computer program stored in the memory 132, and implements any of the image processing methods provided in the embodiments of the present invention when executing the computer program.

[0082] In one embodiment, the processor 131 is configured to run a computer program stored in a memory, and when executing the computer program, to perform the following steps: Acquire multiple first images and acquire a target motion mask, wherein the target motion mask is used to describe whether the image region where each image unit in each first image is located is a moving region or a static region, and the image unit includes one or more pixels; According to the target motion mask, the plurality of first images are subjected to HDR fusion processing to obtain a second image. The HDR fusion processing of the plurality of first images includes: for each image unit, if the image region where the image unit is located is determined to be a moving region according to the target motion mask, HDR fusion processing is performed based on the pixel values ​​of the image unit in a first number of first images; if the image region where the image unit is located is determined to be a static region according to the target motion mask, HDR fusion processing is performed based on the detail information of the image unit in a second number of first images, where the first number is less than the second number.

[0083] In some embodiments, when acquiring multiple first images, the processor 131 is configured to: Acquire the image frame sequence captured by the imaging device of the head-mounted device and the IMU measurement data synchronized with the image frame sequence acquired by the inertial measurement unit of the head-mounted device; The inter-frame rotation matrix of each image frame in the image frame sequence relative to a reference frame in the image frame sequence is determined based on the IMU measurement data. Based on the inter-frame rotation matrix of each image frame in the image frame sequence relative to the reference frame, the extrinsic rotation matrix between the inertial measurement unit and the imaging device, and the intrinsic parameter matrix of the imaging device, the homography transformation matrix corresponding to each image frame is determined. Perspective transformation is performed on each of the image frames by applying their respective homography transformation matrix to obtain the plurality of first images spatially aligned with the reference frame.

[0084] In some embodiments, when acquiring the target motion mask, the processor 131 is configured to: The motion intensity parameters of the head-mounted device are determined based on the IMU measurement data. Based on the homography transformation matrix of each first image relative to a reference image among the plurality of first images, a first motion vector of each image unit in the reference image relative to each first image is determined; Based on the first motion vector of each image unit in the reference image relative to each first image, determine the first motion amplitude of each image unit in the reference image relative to each first image; The target motion mask is generated based on the first motion amplitude and the motion intensity parameter of each image unit in the reference image relative to each first image.

[0085] In some embodiments, when acquiring the target motion mask, the processor 131 is configured to: The motion intensity parameters of the head-mounted device are determined based on the IMU measurement data. Based on the homography transformation matrix of each first image relative to a reference image among the plurality of first images, a first motion vector of each image unit in the reference image relative to each first image is determined; Based on the first motion vector of each image unit in the reference image relative to each first image, determine the first motion amplitude of each image unit in the reference image relative to each first image; Based on a preset optical flow algorithm, a second motion vector is determined for each image unit in the reference image relative to each first image; The second motion amplitude of each image unit in the reference image relative to each first image is determined based on the second motion vector of each image unit in the reference image relative to each first image; Based on the first motion amplitude and the second motion amplitude of each image unit in the reference image relative to each first image, the target motion amplitude of each image unit in the reference image relative to each first image is determined; The target motion mask is generated based on the target motion amplitude and motion intensity parameters of each image unit in the reference image relative to each first image.

[0086] In some embodiments, when the processor 131 performs HDR fusion processing on the plurality of first images according to the target motion mask to obtain a second image, it is configured to: Based on the first strategy, the first number of first images are subjected to HDR fusion processing to obtain a first fused image, wherein the first strategy is used to suppress the afterimage of the moving region. Based on the second strategy, HDR fusion processing is performed on the second number of first images to obtain a second fused image. The exposure parameters of the second number of first images are different. The first strategy is different from the second strategy. The second strategy is used to expand the dynamic range of the static region. Based on the target motion mask, the first fused image and the second fused image are weighted and synthesized to obtain the second image.

[0087] In some embodiments, the first number of first images includes reference images, and the processor 131, when implementing HDR fusion processing on the first number of first images based on a first strategy to obtain a first fused image, is configured to: For each of the first images in the first number of first images, determine the pixel value difference of each image unit in the first image relative to the reference image; The weighting coefficient of each image unit in the first image is determined based on the difference in pixel value between each image unit in the first image and the reference image; For each image unit in the reference image, the target pixel value of the image unit is determined based on the pixel value of the image unit in each of the first images and the corresponding weighting coefficients; The current pixel value of each image unit in the reference image is replaced with the corresponding target pixel value to obtain the first fused image.

[0088] In some embodiments, when the processor 131 performs HDR fusion processing on the second number of first images based on the second strategy to obtain a second fused image, it is configured to: Each of the first images in the second number of first images is decomposed into a multi-scale representation to obtain a multi-scale representation of each first image, wherein the multi-scale representation includes detail information at multiple scale levels. Based on the image quality metric parameters of each image unit in each first image, a fusion weight map corresponding to each first image is generated, wherein the image quality metric parameters are used to describe the confidence level of the image unit in the HDR fusion process; Based on the fusion weight map corresponding to each first image, the multi-scale representation corresponding to each first image is weighted and fused at the multiple scale levels to obtain the target multi-scale representation; The target multi-scale representation is reconstructed at multiple scales to obtain the second fused image.

[0089] In some embodiments, after performing HDR fusion processing on the plurality of first images according to the target motion mask to obtain a second image, the processor 131 is further configured to: A first imaging region and a second imaging region are determined in the second image. The first imaging region is located within the boundary of the original image captured by the camera of the head-mounted device, and the second imaging region is located outside the boundary of the original image captured by the camera of the head-mounted device. Determine the field of view center of the reference image of the head-mounted device in the plurality of first images, and determine a cropping reference center in the first imaging area based on the field of view center; Based on the cropping reference center and the target resolution, the first imaging region is cropped to obtain the target image.

[0090] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the image processing device described above can be referred to the corresponding process in the foregoing image processing method embodiments, and will not be repeated here.

[0091] This invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement any of the image processing methods provided in the specification of this invention.

[0092] The storage medium can be volatile or non-volatile. It can be an internal storage unit of the head-mounted device described in the foregoing embodiments, such as the hard drive or memory of the head-mounted device. Alternatively, it can be an external storage device of the head-mounted device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard.

[0093] Those skilled in the art will understand that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0094] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire multiple first images and acquire a target motion mask, wherein the target motion mask is used to describe whether the image region where each image unit in each first image is located is a moving region or a static region, and the image unit includes one or more pixels; According to the target motion mask, the plurality of first images are subjected to HDR fusion processing to obtain a second image. The HDR fusion processing of the plurality of first images includes: for each image unit, if the image region where the image unit is located is determined to be a moving region according to the target motion mask, HDR fusion processing is performed based on the pixel values ​​of the image unit in a first number of first images; if the image region where the image unit is located is determined to be a static region according to the target motion mask, HDR fusion processing is performed based on the detail information of the image unit in a second number of first images, where the first number is less than the second number.

2. The image processing method according to claim 1, characterized in that, The acquisition of multiple first images includes: Acquire the image frame sequence captured by the imaging device of the head-mounted device and the IMU measurement data synchronized with the image frame sequence acquired by the inertial measurement unit of the head-mounted device; The inter-frame rotation matrix of each image frame in the image frame sequence relative to a reference frame in the image frame sequence is determined based on the IMU measurement data. Based on the inter-frame rotation matrix of each image frame in the image frame sequence relative to the reference frame, the extrinsic rotation matrix between the inertial measurement unit and the imaging device, and the intrinsic parameter matrix of the imaging device, the homography transformation matrix corresponding to each image frame is determined. Perspective transformation is performed on each of the image frames by applying their respective homography transformation matrix to obtain the plurality of first images spatially aligned with the reference frame.

3. The image processing method according to claim 2, characterized in that, The acquisition of the target motion mask includes: The motion intensity parameters of the head-mounted device are determined based on the IMU measurement data. Based on the homography transformation matrix of each first image relative to a reference image among the plurality of first images, a first motion vector of each image unit in the reference image relative to each first image is determined; Based on the first motion vector of each image unit in the reference image relative to each first image, determine the first motion amplitude of each image unit in the reference image relative to each first image; The target motion mask is generated based on the first motion amplitude and the motion intensity parameter of each image unit in the reference image relative to each first image.

4. The image processing method according to claim 3, characterized in that, The method further includes: Based on a preset optical flow algorithm, a second motion vector is determined for each image unit in the reference image relative to each first image; The second motion amplitude of each image unit in the reference image relative to each first image is determined based on the second motion vector of each image unit in the reference image relative to each first image; The step of generating the target motion mask based on the first motion amplitude and the motion intensity parameter of each image unit in the reference image relative to each first image includes: Based on the first motion amplitude and the second motion amplitude of each image unit in the reference image relative to each first image, the target motion amplitude of each image unit in the reference image relative to each first image is determined; The target motion mask is generated based on the target motion amplitude and motion intensity parameters of each image unit in the reference image relative to each first image.

5. The image processing method according to claim 1, characterized in that, The step of performing HDR fusion processing on the plurality of first images according to the target motion mask to obtain a second image includes: Based on the first strategy, the first number of first images are subjected to HDR fusion processing to obtain a first fused image, wherein the first strategy is used to suppress the afterimage of the moving region. Based on the second strategy, HDR fusion processing is performed on the second number of first images to obtain a second fused image. The exposure parameters of the second number of first images are different. The first strategy is different from the second strategy. The second strategy is used to expand the dynamic range of the static region. Based on the target motion mask, the first fused image and the second fused image are weighted and synthesized to obtain the second image.

6. The image processing method according to claim 5, characterized in that, The first number of first images includes reference images, and the step of performing HDR fusion processing on the first number of first images based on a first strategy to obtain a first fused image includes: For each of the first images in the first number of first images, determine the pixel value difference of each image unit in the first image relative to the reference image; The weighting coefficient of each image unit in the first image is determined based on the difference in pixel value between each image unit in the first image and the reference image; For each image unit in the reference image, the target pixel value of the image unit is determined based on the pixel value of the image unit in each of the first images and the corresponding weighting coefficients; The current pixel value of each image unit in the reference image is replaced with the corresponding target pixel value to obtain the first fused image.

7. The image processing method according to claim 5, characterized in that, The step of performing HDR fusion processing on the second number of first images based on the second strategy to obtain a second fused image includes: Each of the first images in the second number of first images is decomposed into a multi-scale representation to obtain a multi-scale representation of each first image, wherein the multi-scale representation includes detail information at multiple scale levels. Based on the image quality metric parameters of each image unit in each first image, a fusion weight map corresponding to each first image is generated, wherein the image quality metric parameters are used to describe the confidence level of the image unit in the HDR fusion process; Based on the fusion weight map corresponding to each first image, the multi-scale representation corresponding to each first image is weighted and fused at the multiple scale levels to obtain the target multi-scale representation; The target multi-scale representation is reconstructed at multiple scales to obtain the second fused image.

8. The image processing method according to any one of claims 1-7, characterized in that, After performing HDR fusion processing on the plurality of first images according to the target motion mask to obtain the second image, the method further includes: A first imaging region and a second imaging region are determined in the second image. The first imaging region is located within the boundary of the original image captured by the camera of the head-mounted device, and the second imaging region is located outside the boundary of the original image captured by the camera of the head-mounted device. Determine the field of view center of the reference image of the head-mounted device in the plurality of first images, and determine a cropping reference center in the first imaging area based on the field of view center; Based on the cropping reference center and the target resolution, the first imaging region is cropped to obtain the target image.

9. An image processing apparatus, characterized in that, The image processing apparatus includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for enabling communication between the processor and the memory, wherein the computer program, when executed by the processor, implements the image processing method as described in any one of claims 1 to 8.

10. A head-mounted device, comprising a camera and an inertial measurement unit, characterized in that, It also includes the image processing apparatus of claim 9.

11. A storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the image processing method according to any one of claims 1 to 8.

Images