Image processing method and apparatus therefor

Abstract

The application discloses an image processing method and device, and belongs to the technical field of image processing. The method comprises the following steps: collecting a first image through a first camera and collecting a second image through a second camera, wherein the field of view angle of the first camera contains the field of view angle of the second camera; processing the first image based on the image features of the first image and the image features of the second image to obtain a third image; wherein the image content of the third image matches the image content of the second image, and the field of view angle corresponding to the third image matches the field of view angle corresponding to the second image; and fusing the first region in the third image and the second image to obtain a target image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and specifically relates to an image processing method and apparatus. Background Technology

[0002] With the rapid development of imaging technology for electronic devices and the ever-increasing demands for image quality in electronic device photography, the lens architecture of electronic devices has gradually upgraded to multi-camera combinations, such as combinations of telephoto and wide-angle lenses, or combinations of telephoto and main lenses. Due to space constraints in miniaturizing electronic devices, lenses must employ multi-refractive / reflective optical structures to reduce volume, significantly increasing the risk of stray light in the optical path. Therefore, stray light suppression has become a key research direction in the field of optical imaging for electronic devices.

[0003] In related technologies, hardware optimization solutions are mainly used to suppress the formation of stray light in lenses. The mainstream methods include optical coatings, light-blocking with light-blocking plates, and ink absorption on the inner wall of the lens. However, hardware stray light suppression technology is constrained by space, optical architecture, process, and cost factors, and may not be able to completely eliminate stray light generated by complex optical paths.

[0004] Therefore, using hardware to suppress stray light will result in insufficient stray light suppression, leading to poor stray light suppression effect. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can improve the suppression of stray light in images.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring a first image through a first camera and acquiring a second image through a second camera, wherein the field of view of the first camera includes the field of view of the second camera; processing the first image based on image features of the first image and image features of the second image to obtain a third image; wherein the image content of the third image is matched with that of the second image, and the field of view of the third image is matched with that of the second image; and fusing a first region in the second image with the third image to obtain a target image.

[0007] Secondly, embodiments of this application provide an image processing apparatus, comprising: an acquisition module and a processing module; the acquisition module is configured to acquire a first image via a first camera and a second image via a second camera, wherein the field of view of the first camera includes the field of view of the second camera; the processing module is configured to process the first image based on the image features of the first image acquired by the acquisition module and the image features of the second image acquired by the acquisition module to obtain a third image; wherein the image content of the third image is matched with that of the second image, and the field of view of the third image is matched with that of the second image; the processing module is further configured to fuse a first region in the second image with the third image to obtain a target image.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, a first image is first acquired using a first camera, and a second image is acquired using a second camera, wherein the field of view of the first camera includes the field of view of the second camera. Then, based on the image features of the first and second images, the first image is processed to obtain a third image. The third image matches the image content of the second image, and the corresponding field of view of the third image matches that of the second image. Finally, the first region in the second image is fused with the third image to obtain the target image. In this solution, by utilizing the overlapping area of ​​the dual cameras of the electronic device, the first region in the second image that may contain stray light is replaced by an image region in the first image with the same field of view, thereby improving the suppression effect of stray light in imaging. Attached Figure Description

[0013] Figure 1This is a flowchart of an image processing method provided in an embodiment of this application;

[0014] Figure 2 This is a schematic diagram illustrating an example of a camera shooting area provided in an embodiment of this application;

[0015] Figure 3 (a) is a schematic diagram of stray light display in a telephoto lens under a main camera and telephoto lens combination provided in an embodiment of this application;

[0016] Figure 3 (b) is a schematic diagram of stray light display in the main camera lens of a main camera and telephoto lens combination provided in an embodiment of this application;

[0017] Figure 4 (a) is a schematic diagram of the imaging display of a telephoto lens in a combination of a main camera and a telephoto lens provided in an embodiment of this application;

[0018] Figure 4 (b) is a schematic diagram of the imaging display of the main camera lens in a combination of a main camera and a telephoto lens provided in an embodiment of this application;

[0019] Figure 4 (c) is a schematic diagram of a target image after fusion processing under the combination of a main camera and a telephoto lens provided in an embodiment of this application;

[0020] Figure 5 This is a flowchart illustrating a specific embodiment of an image processing method provided in this application.

[0021] Figure 6 This is a flowchart illustrating another specific embodiment of the image processing method provided in this application.

[0022] Figure 7 (a) is a schematic diagram of stray light display in a telephoto lens under a main camera and telephoto lens combination provided in an embodiment of this application;

[0023] Figure 7 (b) is a schematic diagram of stray light display in the main camera lens of a main camera and telephoto lens combination provided in an embodiment of this application;

[0024] Figure 8 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0025] Figure 9 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0026] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0027] Figure 11 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0028] Figure 12 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0030] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects. For example, a first object can be one or more, where "more" means at least two. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The terms "at least one" and "at least one of" in this application's specification refer to any one, any two, or a combination of two or more of the included objects. For example, "at least one of a, b, and c" can mean "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple, and multiple means at least two. Similarly, "at least two" means two or more, and its meaning is similar to "at least one". The identifiers in this application are text, symbols, images, etc., used to indicate information, and can use controls or other containers as carriers for displaying information, including but not limited to text identifiers, symbol identifiers, and image identifiers.

[0032] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0033] Lens stray light is a key factor affecting image quality. It is caused by strong light sources inside and outside the lens's field of view, manifesting as ghosting due to multiple reflections and / or refractions from optical surfaces, flare due to diffuse reflection and / or scattering from structural components, flare around strong light sources, and resulting in localized or overall washed-out images and reduced contrast. In critical scenarios such as high dynamic range night scenes, stage performances, backlit portraits, and landscape photography, stray light can severely impact image quality and the user experience.

[0034] As the pursuit of image quality in electronic device photography continues to increase, and optical architecture continues to evolve, the focal length of electronic device cameras has evolved from single fixed focus to multi-camera combinations, especially telephoto lenses, whose equivalent focal length has been continuously upgraded, reaching 240mm or even longer. However, increased focal length means a longer optical path. Within the limited space of electronic devices, the compactness of telephoto lenses often relies on increasing the number of refractive and / or reflective interfaces in the optical architecture. This inevitably introduces more stray light risks. Many of these stray light rays overlap with normal imaging light paths and cannot be directly blocked or reduced by using light-shielding structures. Therefore, the presence of stray light will seriously affect the captured image.

[0035] Currently, effective solutions for stray light mainly include optical coatings, light-blocking plates, and ink absorption. However, these solutions primarily rely on hardware blocking, often accompanied by increased costs, smaller apertures, and reduced relative illumination. More problematic is that for stray light generated by complex refraction and / or reflection in some lenses, the effectiveness of these techniques is limited; some stray light cannot be completely eliminated or even significantly reduced.

[0036] Optical coating is a fundamental method for reducing stray light, achieved by depositing multiple thin films on the lens surface to lower interface reflectivity. However, this technology is already widely used in the lenses of electronic devices, significantly reducing interface reflectivity, and cannot be further reduced through technological upgrades. This is especially true for lenses of electronic devices operating in the visible light band, where reducing reflectivity by orders of magnitude across a wide spectral range is extremely difficult. Therefore, in practice, it is often difficult to completely avoid stray light caused by strong light sources; furthermore, coating technology often comes with increased costs or reliability risks.

[0037] Secondly, using structural components such as light-shielding plates is another common method to prevent stray light from reaching the sensor by placing blocking structures at specific locations. However, the placement of light-shielding structures can be severely limited within the compact space of electronic devices; moreover, light-shielding components often block light from parts of the edge or even the center of the field of view simultaneously, resulting in a reduction in relative illumination and aperture; more importantly, as mentioned earlier, the light paths of some stray light overlap significantly or even coincide with those of normal imaging light, and this type of stray light cannot be fundamentally eliminated by light-shielding methods.

[0038] In addition, ink absorption, which reduces reflectivity by coating the inner wall of the lens with a light-absorbing material to suppress stray light, is also a conventional method. However, further reducing the reflectivity of the ink coating material based on current ink technology is quite difficult, and the stability, consistency, and material reliability of the process are also challenges. Uneven coating can also lead to new stray light problems. More importantly, while ink absorption has a certain effect on stray light incident at specific angles, it cannot fundamentally change the stray light path formed by multi-interface reflections in complex optical structures.

[0039] In summary, the aforementioned hardware methods are limited by multiple factors such as optical architecture, cost, and space, and will always face the problem that some stray light cannot be fundamentally eliminated.

[0040] This application provides an image processing method that leverages the inherent differences in characteristics of a dual-camera system to solve the stray light problem that is difficult to eliminate with a single lens. Its core improvement lies in the discovery and application of the principle of stray light complementarity in multi-camera systems: due to differences in optical architecture, different lenses often exhibit different stray light characteristics for the same incident light source. For example, severe flares or ghosting caused by multiple refractions and reflections in a telephoto lens may not exist in the corresponding imaging area of ​​the main camera. Based on this phenomenon, this application utilizes the overlapping area of ​​the dual-camera field of view to register and replace stray light present in the image of one camera with a "clean" image of the corresponding area from the other camera, thereby effectively eliminating stray light that is difficult to handle using traditional methods.

[0041] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0042] The image processing method provided in this application can be applied to shooting scenarios of electronic devices. For example, shooting street scenes, landscapes, people, or animals.

[0043] It should be noted that the above scenarios are merely illustrative examples of some possible applications of the embodiments of this application. In actual implementation, the embodiments of this application can also be applied to any possible scenarios such as electronic device recording, monitoring, or image processing. The embodiments of this application are not limited here.

[0044] Based on the scenario described above in the embodiments of this application, the image processing method provided in this application first acquires a first image using a first camera and a second image using a second camera, wherein the field of view of the first camera includes the field of view of the second camera; then, based on the image features of the first image and the image features of the second image, the first image is processed to obtain a third image; the third image matches the image content of the second image, and the field of view of the third image matches the field of view of the second image; finally, the first region in the second image is fused with the third image to obtain the target image. In this solution, by utilizing the overlapping area of ​​the dual cameras of the electronic device, the first region in the second image that may contain stray light is replaced by an image region in the first image with the same field of view, thereby improving the suppression effect of stray light in imaging.

[0045] The image processing method provided in this application is executed by an image processing device, which can be an electronic device, or a functional module or entity within an electronic device. This application does not limit the specific implementation of this method. The following will use an electronic device as an example to illustrate the image processing method provided in this application.

[0046] This application provides an image processing method. Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application is shown. Figure 1 As shown, the image processing method provided in this application embodiment may include the following steps 201 to 203.

[0047] Step 201: The electronic device acquires a first image through the first camera and a second image through the second camera.

[0048] In some embodiments of this application, the field of view of the first camera includes the field of view of the second camera.

[0049] For example, the field of view of the first camera completely covers the field of view of the second camera.

[0050] For example, the field of view (FOV) of the first camera is 85° and the FOV of the second camera is 10°.

[0051] In some embodiments of this application, the focal length of the first camera is smaller than that of the second camera.

[0052] In some embodiments of this application, the first camera is a main camera, which has the imaging characteristics of a large field of view and wide coverage.

[0053] In some embodiments of this application, the first camera described above is used to capture the overall scene and obtain a complete scene image.

[0054] For example, the first camera mentioned above is a wide-angle camera or an ultra-wide-angle camera, etc.

[0055] In some embodiments of this application, the second camera is a secondary camera with imaging features such as high pixel count and high resolution.

[0056] In some embodiments of this application, the second camera described above is used to capture detailed images of a local scene, that is, to capture clear detail images of a specific area.

[0057] For example, the second camera mentioned above is a telephoto camera or a periscope telephoto camera, etc.

[0058] It is understood that the image processing method provided in this application mainly targets the combination of a main camera and a telephoto lens. Taking stray light caused by light sources within the field of view (FOV) of a telephoto module as an example, it solves the problems of flare and ghosting that are difficult to eliminate by traditional pure hardware means, especially in periscope telephoto lenses. This application has the following requirements for the configuration of the dual-camera system, namely the first camera and the second camera: First, the field of view of the main camera (i.e., the first camera) and the telephoto lens (i.e., the second camera) must overlap, so that the same scene can be imaged by both cameras simultaneously. This overlapping area covers areas in the telephoto image that may contain stray light. Second, the focal length of the main camera is significantly smaller than that of the telephoto lens, so that the two lenses can have sufficient differences in their optical architecture, thereby producing different stray light performances for the same incident light source. Third, the images captured by the dual-camera system must maintain a certain consistency in brightness and color response to facilitate subsequent registration and fusion processing.

[0059] For example, taking the first camera as the main camera with a field of view (FOV) of 85° and the second camera as a telephoto lens with a FOV of 10°, the relationship between their FOVs and the image received by the final image sensor is illustrated as follows: Figure 2 As shown. The images captured by the two lenses are denoted as C1 and C2. Region S is the area containing stray light to be eliminated within the telephoto lens. This stray light is generated by multiple reflections within the telephoto lens module from a light source with an incident angle of 3°, and is then formed on the image sensor, as shown below. Figure 3 The stray light pattern shown in (a) is as follows; while the stray light pattern formed by non-imaging rays after this light source passes through the main camera lens module is as follows. Figure 3 As shown in (b): Thus, it can be seen that for light sources at this angle of incidence, the number of non-imaging rays reaching the image sensor in the main camera lens is extremely small or even negligible, and therefore, stray light patterns are basically not formed.

[0060] In some embodiments of this application, the first image described above is a complete image captured by the first camera.

[0061] In some embodiments of this application, the second image described above is a partial image captured by a second camera.

[0062] For example, the image content of the second image is a portion of the image content of the first image.

[0063] In some embodiments of this application, the electronic device simultaneously captures its corresponding first image and second image through a first camera and a second camera.

[0064] In some embodiments of this application, the electronic device uses a first camera as the main camera and a second camera as the slave camera to capture images of the same scene at the same time, based on mechanisms such as frame synchronization.

[0065] For example, when an electronic device uses a second camera to take pictures, if a strong light source is detected, the first and second cameras of the electronic device are simultaneously triggered to acquire images.

[0066] It is understandable that the detection of light sources can be achieved by recognizing the image content or by determining the overexposure and saturation of the local RGB intensity of the image.

[0067] For example, the aforementioned synchronous acquisition refers to the main camera lens and the telephoto lens, that is, the first camera and the second camera, completing exposure at the same moment or within a very short time interval to acquire dual-camera images of the same scene, and the acquired images are respectively as follows: Figure 4 (a) and Figure 4 As shown in (b) of the diagram.

[0068] For example, to ensure that the dual cameras capture the scene at the same moment and avoid subsequent registration failure due to scene changes, the electronic device derives the pixel clocks based on the same system clock and introduces a hardware frame synchronization mechanism. Hardware-level linkage between the dual cameras is achieved through hardware trigger pins (such as GPIO / FSYNC), enabling the telephoto lens to listen to the trigger signal of the main camera lens in real time, thereby ensuring that the image sensors connected to the two cameras achieve simultaneous exposure and data reading at the microsecond level.

[0069] For example, during the image acquisition stage, white balance correction, depigmentation, and other image processing can be performed on the first and second images to obtain standard RGB images, which will facilitate subsequent image analysis and processing.

[0070] Step 202: The electronic device processes the first image based on the image features of the first image and the image features of the second image to obtain the third image.

[0071] In some embodiments of this application, the image content of the third image is matched with the image content of the second image.

[0072] For example, the above image content matching means that the image content such as the target object and scene elements contained in the third image corresponds to the image content in the second image.

[0073] In some embodiments of this application, the field of view corresponding to the third image is matched with the field of view corresponding to the second image.

[0074] For example, the above field of view matching means that the field of view range of the third image is the same as that of the second image.

[0075] It is understandable that due to the different field of view and positions of the main camera and the telephoto lens, i.e. the first camera and the second camera mentioned above, there is a parallax. The pixel positions of the same scene in the dual-camera images have a corresponding relationship, and it is necessary to establish a mapping between the dual-camera images through registration.

[0076] For example, the electronic device uses the field of view parameters of the second image as a reference to perform field of view mapping and boundary adaptation processing on the first image, and performs cropping and adaptation correction on the first image according to the field of view angle, framing ratio and imaging range of the second camera, so that the effective imaging field of view, screen angle and framing boundary size of the final output of the third image are consistent with the second image.

[0077] For example, the electronic device can perform distortion correction on the main camera image, i.e., the first image mentioned above, based on the intrinsic parameters of the two cameras calibrated at the factory, such as focal length, principal point coordinates, distortion coefficients, etc., and extrinsic parameters, such as relative positional relationship, rotation matrix, etc. Then, the corrected main camera image is cropped and mapped to the telephoto view of the second image through geometric transformation to obtain the registered main camera image, i.e., the third image mentioned above.

[0078] Step 203: The electronic device fuses the first region in the second image with the third image to obtain the target image.

[0079] In some embodiments of this application, the first region described above may be a stray light region in the second image.

[0080] In some embodiments of this application, the aforementioned stray light region is the region where stray light exists.

[0081] In some embodiments of this application, the aforementioned stray light region is an abnormal pixel region in the image formed by strong ambient light, lens reflection, light refraction, and scattering interference.

[0082] In some embodiments of this application, the aforementioned stray light is interference light generated by non-target imaging light sources during the shooting process, such as the light source of a street lamp projected onto the camera's photosensitive element. This light does not belong to the imaging light of the scene target itself and is the main interference factor causing image overexposure, color shift, image blurring, and loss of detail.

[0083] In some embodiments of this application, the electronic device can determine the stray light region in the image based on the imaging characteristics of stray light, the brightness value of image pixels, etc.

[0084] In some embodiments of this application, the electronic device uses a fusion algorithm to organically combine the stray light region in the second image, namely the high-frequency texture detail features of the first region, with the mid-to-low frequency structure and color features of the corresponding region in the first image, thereby removing the stray light distribution portion in the second image that is relatively low-frequency and has a color distribution significantly different from that of the third image region.

[0085] For example, the above-mentioned fusion algorithm can be an Image Fusion Convolutional Neural Network (IFCNN), a Unified Unsupervised Image Fusion Network (U2Fusion), an Enhanced Super-Resolution Generative Adversarial Network (ESRGAN), etc.; or the above-mentioned fusion algorithm can also be an image fusion algorithm based on multi-scale decomposition, such as Laplacian Pyramid Blending, Wavelet Fusion, and Guided Filtering Fusion.

[0086] It should be noted that, to ensure the naturalness of the fusion of the stray light regions in the third and second images and avoid obvious seams or artifacts, the following strategies can be adopted: calculate the edge consistency between the stray light regions and the surrounding normal regions in the second image to ensure a smooth edge transition after replacement; if necessary, multi-scale fusion or Poisson fusion techniques can be used to further optimize the visual effect of the image regions in the target image other than the stray light regions in the second image and the seams in the third image. Multi-scale fusion refers to decomposing the image into multiple scale layers and fusing and reconstructing at different scale layers; Poisson fusion refers to performing fusion processing in the gradient domain to maintain edge continuity.

[0087] For example, after the electronic device identifies the stray light area, it replaces the image pixel values ​​of the stray light area in the telephoto image (i.e., the image pixel values ​​of the stray light area in the second image) with the image pixel values ​​of the main camera's mapped area (i.e., the image pixel values ​​of the third image mentioned above), thus completing the fusion output effect as shown below. Figure 4 As shown in (c) in the figure.

[0088] It should be noted that since the images from the telephoto lens and the main camera, i.e., the first image and the second image, are generated by image sensors with different resolutions capturing scenes within different fields of view (FOVs), the stray light areas to be fused and replaced correspond to different resolutions in the images from the two modules. This is especially true when the resolutions differ significantly. For example, if the resolution of the area without stray light replacement is significantly lower than the resolution of the area being replaced, direct interpolation and pixel replacement will lead to a loss of image details. Therefore, it should be explained that the fusion processing, or replacement operation, in this embodiment does not involve directly replacing pixels between the two areas. Instead, it is achieved through a fusion algorithm, similar to the telephoto and main camera fusion, main camera and wide-angle camera fusion, RGB and multispectral fusion, and monochrome and color fusion commonly found in existing products. This avoids the shortcomings of each camera and combines their respective advantages.

[0089] In the image processing method provided in this application embodiment, a first image is first acquired using a first camera, and a second image is acquired using a second camera, wherein the field of view of the first camera includes the field of view of the second camera. Then, based on the image features of the first and second images, the first image is processed to obtain a third image. The third image matches the image content of the second image, and the corresponding field of view of the third image matches that of the second image. Finally, the first region in the second image is fused with the third image to obtain the target image. In this solution, by utilizing the overlapping area of ​​the dual cameras of the electronic device, the first region in the second image that may contain stray light is replaced by an image region in the first image with the same field of view, thereby improving the suppression effect of stray light in imaging.

[0090] Optionally, in some embodiments of this application, the above step 202, "the electronic device processes the first image based on the image features of the first image and the image features of the second image to obtain the third image," can be specifically implemented through the following steps 202a to 202d.

[0091] Step 202a: The electronic device acquires the first image feature information of the first image and the second image feature information of the second image.

[0092] In some embodiments of this application, the aforementioned first image feature information is used to characterize feature points in the first image.

[0093] In some embodiments of this application, the aforementioned second image feature information is used to characterize feature points in the second image.

[0094] For example, the image feature information mentioned above may include, but is not limited to, the image content, pixel values, brightness values, contrast values, and other feature information of the image.

[0095] For example, the aforementioned feature information includes, but is not limited to, Oriented FAST and Rotated BRIEF (ORB) and Scale-Invariant Feature Transform (SIFT).

[0096] Step 202b: The electronic device matches the first image feature information and the second image feature information to obtain pixel matching point pairs.

[0097] In some embodiments of this application, the aforementioned pixel matching point pair includes pixels in the first image and pixels in the second image that are at the same position as the first pixel.

[0098] For example, the electronic device can perform matching based on the feature information of the main camera image, i.e. the first image mentioned above, and the telephoto image, i.e. the second image mentioned above, to construct an initial matching point pair and eliminate mismatched point pairs.

[0099] Step 202c: The electronic device calculates the transformation matrix that maps the field of view of the first image to the field of view of the second image based on pixel matching point pairs.

[0100] In some embodiments of this application, the transformation matrix described above is a 2D homography transformation matrix.

[0101] For example, the matrix parameters of the above transformation matrix include, but are not limited to, at least one of the following: rotation, translation, scaling and perspective transformation components.

[0102] For example, the electronic device calculates the two-dimensional (2D) homography transformation matrix from the main camera to the telephoto field of view, i.e. from the first camera to the second camera field of view, based on the aforementioned pixel matching point pairs.

[0103] Step 202d: The electronic device processes the first image based on the transformation matrix to obtain the third image.

[0104] In some embodiments of this application, the electronic device processes the first image based on the above-mentioned transformation matrix to determine the target pixel region in the first image. The target pixel region is the image pixel region mapped by the field of view of the second camera in the first image. Then, the image region in the first image excluding the target pixel region is cropped to obtain the above-mentioned third image.

[0105] Optionally, in some embodiments of this application, step 202d can be implemented by steps 202d1 and 202d2.

[0106] Step 202d1: Based on the field of view and resolution information of the second image, the electronic device performs reverse coordinate mapping on the transformation matrix to determine the target area of ​​the first image under the field of view of the second image.

[0107] In some embodiments of this application, the target area is the image pixel area mapped by the field of view of the second camera in the first image.

[0108] For example, the electronic device uses the transformation matrix to perform reverse coordinate mapping based on the field of view and resolution of the telephoto image, i.e. the second image, to determine the target effective pixel area of ​​the main camera image, i.e. the first image, under the telephoto view of the second camera, i.e. the target area.

[0109] Step 202d2: The electronic device crops the image outside the target area in the first image to obtain the third image.

[0110] For example, the electronic device resamples or performs super-resolution processing on the pixels in the target area, while cropping and discarding pixels in the first image that exceed the boundary of the target area, to obtain a registered main camera image that is aligned with the resolution and viewing angle of the telephoto image, i.e., the third image mentioned above.

[0111] Thus, by improving registration accuracy through the above methods to ensure stray light detection and fusion effects, sub-pixel level accuracy can be achieved, thereby avoiding misalignment between the replacement area and the surrounding area, resulting in obvious seams, and thus improving the image imaging effect.

[0112] Optionally, in some embodiments of this application, the electronic device can locate the stray light region in different ways, such as determining the stray light region by simulating or predicting the imaging of the light source, determining the stray light region based on stray light characteristic information, or determining the stray light region based on the image brightness value or image parameters of the acquired image, etc. The process of determining the stray light region in the second image is described below with four possible embodiments.

[0113] First possible implementation:

[0114] In some embodiments of this application, the electronic device can simulate or calibrate the light source based on parameters such as the position and incident angle of the light source to generate stray light imaging and determine the stray light region.

[0115] For example, if the light source that generates stray light is located within the FOV of the main camera lens, i.e. the first camera mentioned above, the position and angle of incidence of the light source can be located through the image of the main camera lens, and the stray light area corresponding to the light source in the telephoto image, i.e. the second image mentioned above, can be determined based on the calibration data of simulation / actual measurement.

[0116] For example, if the light source that generates stray light is located within the telephoto lens, i.e., the FOV of the second camera mentioned above, the position and incident angle of the light source can be located directly through the image of the telephoto lens, and the stray light area corresponding to the light source in the telephoto image, i.e., the second image mentioned above, can be determined based on the calibration data of simulation / actual measurement.

[0117] In this way, electronic devices can simulate the imaging of stray light by simulating or calibrating actual data to determine the stray light area, thereby ensuring the accuracy of stray light suppression.

[0118] The second possible implementation:

[0119] Optionally, in some embodiments of this application, before step 203 above, the image processing method provided in the embodiments of this application further includes steps 301 and 302.

[0120] Step 301: The electronic device acquires stray light feature information in the second image.

[0121] In some embodiments of this application, the above-mentioned stray light characteristic information is used to characterize the imaging features of stray light.

[0122] For example, the stray light feature information mentioned above includes flare features or ghosting features, etc.

[0123] For example, different stray lights have different stray light characteristic information.

[0124] Step 302: The electronic device determines the stray light region in the second image based on the stray light feature information and the image features of the second image.

[0125] In some embodiments of this application, the image features of the second image are image parameters or image content and other feature information of the second image.

[0126] In some embodiments of this application, image parameters of a second image are obtained, and then the image parameters of the second image are calculated to obtain the calculated image parameters and the conditions that the image parameters corresponding to different stray light feature information must meet, so as to determine whether there is a pixel region in the current second image that meets these conditions. If it exists, the pixel region is determined as a stray light region. If it does not exist, it means that there is no stray light region in the second image.

[0127] The process of determining the stray light region in the second image is explained below using two different types of stray light feature information.

[0128] Example 1: Stray light feature information is flare feature.

[0129] For example, flare features appear as bright stripes extending along the direction of the light source, exhibiting directionality and periodicity, and can be detected through directional filtering or frequency domain analysis.

[0130] For example, the electronic device uses the Sobel operator (Sobel) to extract the image gradient of the second image, obtaining the horizontal gradient G. x and vertical gradient G y And calculate the gradient direction angle θ = arctan(G x / G y Then, based on the gradient direction angle θ, the local region in the second image is statistically analyzed, for example, the direction angle histogram within a 100×100 window. When the direction angle histogram in the second image has a significant main direction peak, such as a peak percentage > 40%, and the gradient magnitude in that direction is greater than a set threshold, such as 80, then the corresponding window region is determined to be a flare stray light region, that is, the stray light region in the second image is the flare stray light region.

[0131] Example 2: The stray light feature information is a ghost shadow feature.

[0132] For example, ghosting features are characterized by light spots that are symmetrical to the position of the light source, have regular shapes, and are related to the brightness of the light source. They can be detected through symmetry analysis or template matching.

[0133] For example, the electronic device locates the center point P0 of a light source in the image whose brightness exceeds a threshold, such as 220, and calculates its symmetrical point P1 about the center of the image with P0 as the center. Then it detects whether there is a situation where the brightness of the area near P1 is higher than that of the background. If there is, the area is determined to be a ghost image stray light area, that is, the stray light area in the second image is the ghost image stray light area.

[0134] For example, the aforementioned nearby region can be a region with a radius of r. The value of r is calculated iteratively using the energy within the circle, showing the curve of its change with respect to r, until the rate of change is less than a threshold such as 5%.

[0135] For example, the situation where the brightness is higher than the background can be that there are two peaks with an intensity ratio greater than 2 in the brightness histogram of the nearby area.

[0136] In this way, electronic devices can determine stray light areas by analyzing the stray light characteristics of different stray lights in the telephoto image and combining them with the image features of the second image, thereby ensuring the accuracy of stray light suppression.

[0137] A third possible implementation:

[0138] Optionally, in some embodiments of this application, before step 203 above, the image processing method provided in the embodiments of this application further includes steps 401 and 402.

[0139] Step 401: The electronic device detects the brightness value of each pixel in each pixel block in the second image.

[0140] In some embodiments of this application, a pixel block includes at least two pixels.

[0141] In some embodiments of this application, the second image of the electronic device is divided into blocks, and each pixel block includes at least two pixels, and the brightness value of each pixel in each pixel block is obtained respectively.

[0142] Step 402: The electronic device determines the area where the first pixel block is located as the stray light area in the second image.

[0143] In some embodiments of this application, the electronic device calculates the average brightness value, the maximum brightness value, and the brightness variance value between each pixel block and its neighboring pixel blocks based on the brightness value of each pixel in each pixel block.

[0144] For example, the above average brightness value is the average brightness value of each pixel in a pixel block.

[0145] For example, the maximum brightness value mentioned above is the brightness value of a pixel in a pixel block, which is the maximum value among all brightness values.

[0146] The first pixel block is at least one of the following:

[0147] The pixel blocks in the second image whose maximum brightness value is greater than the second preset threshold;

[0148] The second image contains pixel blocks whose average brightness value is greater than the third preset threshold and whose brightness variance between adjacent pixel blocks is greater than the fourth preset threshold.

[0149] In some embodiments of this application, the aforementioned second preset threshold may be the default value of the electronic device or a custom value.

[0150] In some embodiments of this application, the aforementioned third preset threshold may be the default value of the electronic device or a custom value.

[0151] In some embodiments of this application, the aforementioned fourth preset threshold may be the default value of the electronic device or a custom value.

[0152] For example, the electronic device determines stray light regions based on the detection of a brightness threshold: the telephoto image, the second image mentioned above, is divided into blocks, such as into a 100×100 pixel grid, and the average brightness value L of each sub-block, i.e., the pixel block, is calculated. avg and maximum brightness value L max If certain sub-blocks satisfy L max Greater than the second preset threshold, such as 200, or L avg If the value is greater than the third preset threshold, such as 150, and the brightness variance of adjacent sub-blocks is greater than the fourth preset threshold, such as 500, then the area where the sub-block is located is marked as a suspected stray light area.

[0153] In this way, electronic devices can determine stray light areas by analyzing the brightness values ​​of different pixel blocks in the telephoto image, thereby ensuring the accuracy of stray light suppression.

[0154] A third possible implementation:

[0155] In this embodiment, there is no light source within the FOV of the second camera. For example, a strong light source may be located outside the field of view of the telephoto lens. Light enters the sensor through abnormal paths such as the lens edge, the inner wall of the lens barrel, or structural components, creating stray light in the edges or specific areas of the second image captured by the second camera, such as the telephoto image. However, since the stray light source itself is not within the FOV of the telephoto lens, the light source itself will not appear directly in the telephoto image. Therefore, the two possible embodiments described above differ. In this case, the electronic device may not be able to locate the corresponding stray light area based on pre-simulation / actual calibration by locating the light source position and incident angle in the telephoto image. To address this, this embodiment can locate stray light areas with abnormal brightness by comparing the images between the two cameras, or directly record areas known to have stray light risk as the desired target stray light area.

[0156] Optionally, in some embodiments of this application, before step 203 above, the image processing method provided in the embodiments of this application further includes steps 501 and 502.

[0157] Step 501: The electronic device detects the brightness value of each pixel in the third image and the second image.

[0158] In some embodiments of this application, the electronic device acquires the brightness value of each pixel in the third image and the brightness value of each pixel in the second image.

[0159] Step 502: The electronic device determines the area where the first pixel is located as the stray light area in the second image.

[0160] In some embodiments of this application, the first pixel is a pixel whose difference between the brightness value in the second image and the brightness value in the third image is greater than a first preset threshold.

[0161] In some embodiments of this application, the first preset threshold is either the default value of the electronic device or a custom value.

[0162] For example, the electronic device compares the corrected and cropped main camera image (i.e., the third image) with the telephoto image to match the telephoto image (i.e., the second image mentioned above), detects areas in the telephoto image whose brightness or contrast is significantly higher than that of the matched main camera image, and locates these areas as target stray light areas.

[0163] It should be noted that in this embodiment, the brightness value of each pixel can be obtained as well as the contrast value of each pixel, and this application does not impose any limitation.

[0164] In this way, electronic devices can determine stray light areas by comparing the matched main camera image and the telephoto image when there is no strong light source in the image, thus ensuring the accuracy of stray light suppression.

[0165] It is understood that the effective operation of the embodiments of this application is contingent upon the fulfillment of the stray light complementarity condition. Specifically, when a telephoto lens produces flare or ghosting due to multiple refractions and reflections, the same light source does not produce stray light or produces significantly less stray light in the corresponding imaging area of ​​the main camera. This condition is directly related to the differences in the optical architecture of dual cameras: telephoto lenses typically have a large number of lenses, long optical paths, and complex reflective interfaces, while the main camera, although also having a large number of lenses, has a relatively compact structure. The stray light generation mechanisms and manifestations of the two are significantly different.

[0166] This application provides an image processing method. Figure 5 A flowchart of an image processing method provided in an embodiment of this application is shown.

[0167] The image processing method provided in this application embodiment will be described exemplarily using the combination of a first camera and a second camera as the main camera and a telephoto lens as an example. In this embodiment, the strong light source is located in the image captured by the telephoto lens.

[0168] like Figure 5 As shown, the image processing method provided in this application embodiment may include the following steps A1 to A6.

[0169] Step A1: The electronic device locates the strong light source in the telephoto image.

[0170] For example, the electronic device locates a strong light source appearing in the telephoto view in the shooting preview window.

[0171] Step A2: The electronic device triggers simultaneous acquisition by both cameras.

[0172] For example, the electronic device simultaneously controls the main camera and the telephoto lens to capture images, namely the first image and the second image mentioned above.

[0173] Step A3: The electronic device determines the stray light region, namely the first region mentioned above.

[0174] For example, an electronic device determines stray light regions based on information such as the location of a strong light source, the brightness of the light source, image parameters, and image features.

[0175] Step A4: The electronic device corrects, registers, and crops the main camera image captured by the main camera to obtain the registered main camera image, which is the third image mentioned above.

[0176] Step A5: The electronic device performs a fusion and replacement operation on the registered main camera image and the stray light area.

[0177] Step A6: The electronic device outputs the fusion result, which is the target image mentioned above.

[0178] Thus, this embodiment utilizes the complementary nature of stray light from the main camera and the telephoto lens to replace stray light-affected areas with truly stray light-free image areas, achieving the following technical effects:

[0179] Effectively eliminates stray light in telephoto lenses: It can fundamentally eliminate flare and ghosting caused by multiple refractions and reflections in telephoto lenses;

[0180] No impact on light intake: This embodiment eliminates stray light through image fusion, eliminating the need for additional light-shielding structures, thus avoiding blocking imaging light and preventing a reduction in the module's light intake and aperture.

[0181] No new processes or costs are required: This embodiment makes full use of the existing hardware configuration and information of the dual-camera system, without relying on additional module components or special processes, and will not cause any increase in costs;

[0182] The effect is natural and realistic: The fusion replacement uses real images captured by another camera in the same scene, rather than estimation results based on prior knowledge. Therefore, the fused image has complete details and a natural effect, without the trace residue or detail distortion found in traditional estimation methods.

[0183] It should be noted that the execution order of steps A2 and A3 described above is not limited in this embodiment. For example, step A2 can be executed first, followed by step A3; or step A3 can be executed first, followed by step A2. Figure 4This is illustrated by taking the example of executing step A2 first, and then step A3.

[0184] This application provides another image processing method. Figure 6 A flowchart of another image processing method provided in an embodiment of this application is shown.

[0185] The image processing method provided in this application embodiment will be described exemplarily below using the combination of a first camera and a second camera as the main camera and a telephoto lens as an example. In this embodiment, the strong light source is located outside the frame captured by the telephoto lens.

[0186] It is understood that, in this embodiment, the telephoto lens has a field of view (FOV) of 10° and the main camera lens has a FOV of 85°, meaning the telephoto lens's field of view is smaller than the main camera's. The relationship between their FOVs and the final image received by the image sensor is the same as in the previous embodiment; please refer to [the previous embodiment for details]. Figure 1 When a strong light source is located outside the field of view (FOV) of a telephoto lens, light can still enter the sensor through abnormal paths such as reflections between lenses, scattering at the lens edges, and reflections from the inner walls of the lens barrel, creating stray light patterns in the telephoto image, such as... Figure 7 As shown in (a) above. However, when the same light source passes through the main camera module, due to the difference in field of view between the main camera and the telephoto lens, the light source may be located outside the main camera's FOV, or even if it is within the FOV, its non-imaging ray path is significantly different from that of the telephoto lens, resulting in a noticeably lighter or even non-existent stray light pattern, such as... Figure 7 As shown in (b) of the diagram.

[0187] like Figure 6 As shown, the image processing method provided in this application embodiment may include the following steps B1 to B5.

[0188] Step B1: The electronic device triggers simultaneous acquisition by both cameras.

[0189] For example, the electronic device simultaneously controls the main camera and the telephoto lens to capture images, namely the first image and the second image mentioned above.

[0190] Step B2: The electronic device identifies the stray light area.

[0191] For example, the electronic device may determine the stray light region by the following steps B2-1, B2-2 or B2-3.

[0192] Step B2-1: The electronic device compares the main camera image and the telephoto image to determine the stray light area.

[0193] For example, an electronic device identifies stray light areas by locating areas with abnormal brightness and / or contrast.

[0194] Step B2-2: The electronic device locates the stray light area of ​​the strong light source appearing in the main camera image in the shooting preview window.

[0195] For example, an electronic device determines stray light regions based on information such as the location of a strong light source, the brightness of the light source, image parameters, and image features.

[0196] Step B2-3: The electronic device directly records the area of ​​stray light risk to be examined.

[0197] Step B3: The electronic device corrects, registers, and crops the main camera image captured by the main camera to obtain the registered main camera image, which is the third image mentioned above.

[0198] Step B4: The electronic device performs a fusion and replacement operation on the registered main camera image and the stray light area.

[0199] Step B5: The electronic device outputs the fusion result, which is the target image mentioned above.

[0200] It should be noted that the execution order of steps B1 and B2 described above is not limited in this embodiment. For example, step B1 can be executed first, followed by step B2; or step B2 can be executed first, followed by step B1. Figure 5 This example illustrates the process of executing step B1 first, followed by step B2.

[0201] Thus, this embodiment utilizes the complementary nature of stray light from the main camera and the telephoto lens, replacing stray light-affected areas with truly stray light-free image areas, achieving the same technical effect as steps A1 to A6 described above. Furthermore, this embodiment can resolve the stray light effects caused by light sources outside the field of view (FOV) of the telephoto lens.

[0202] It should be noted that the steps in all embodiments involved in this application, namely the aforementioned detection, recognition, registration, and fusion operations, can also be implemented by training one or more network algorithms. Optional network architectures include U-Net, DeepLab Version 3 Plus (DeepLabV3+), IFCNN, and other convolutional neural network architectures, or end-to-end architectures based on Generative Adversarial Networks (GANs). The training data consists of batch simulation data, i.e., simulated distribution of stray light superimposed on real scene images, such as 10,000 pairs, and real-world image data with manually labeled stray light regions, i.e., real image pairs containing known stray light acquired using a dual-camera system in a laboratory environment, such as 2,000 pairs. The loss function is designed as a weighted combination of multi-task losses, which may include segmentation losses (such as binary cross-entropy loss, used for training stray light region detection), fusion losses (such as L1 norm loss, used to constrain the pixel-level difference between the fused result and the real stray light-free image), and perceptual losses (such as calculating the Euclidean distance between the fused result based on multi-layer feature extraction of a deep convolutional network (Visual Geometry Group Network, VGG) and the feature maps of the real image, used to ensure visual consistency of the fused result). The weighted sum of these three losses is used as the total loss, and the network parameters are trained through backpropagation.

[0203] It is worth noting that the implementation of the stray light complementarity principle in this application embodiment requires the following prerequisites: First, the field of view of the two cameras overlaps, so that the same scene area can be imaged by the two cameras simultaneously; Second, the sensor area where the stray light is located is within the range corresponding to the overlapping area of ​​the field of view of the two cameras; Third, and most importantly, the light source that causes stray light in one camera does not produce stray light or produces significantly less stray light in the corresponding area of ​​the other camera; Fourth, in order to ensure the registration and fusion effect, the dual-camera images need to meet certain consistency requirements in terms of parallax, brightness, and color.

[0204] It should be noted that, in addition to the exemplified implementation methods, different lens combinations can also be used, such as telephoto + wide-angle, or main camera + wide-angle, etc.; in addition to the case mentioned above where the normal image in the large FOV lens is replaced with the stray image in the small FOV lens, the normal image in the small FOV lens can also be replaced with the stray image in the large FOV lens, provided that the stray complementarity condition is met and the stray area in the large FOV lens is within the image of the small FOV lens; besides the combination of small FOV + large FOV, it is also possible to handle the case where the FOV does not meet the inclusion relationship but only meets the partial overlap relationship, provided that the stray area to be solved is within the FOV overlap area and the two lenses meet the stray complementarity condition within this area.

[0205] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0206] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0207] Figure 8 A schematic diagram of a possible structure of the image processing apparatus involved in an embodiment of this application is shown. For example... Figure 8 As shown, the image processing device 700 may include an acquisition module 701 and a processing module 702.

[0208] The acquisition module 701 is used to acquire a first image through a first camera and a second image through a second camera, wherein the field of view of the first camera includes the field of view of the second camera. The processing module 702 is used to process the first image based on the image features of the first image acquired by the acquisition module 701 and the image features of the second image acquired by the acquisition module 701 to obtain a third image. The third image matches the image content of the second image and matches the field of view of the third image. The processing module 702 is also used to fuse a first region in the second image acquired by the acquisition module 701 with the third image to obtain a target image.

[0209] In one possible implementation, the processing module 702 described above is specifically used for:

[0210] Obtain the first image feature information of the first image and the second image feature information of the second image;

[0211] The first image feature information and the second image feature information are matched to obtain pixel matching point pairs. The pixel matching point pairs include the pixels in the first image and the pixels in the second image that are at the same position as the first pixel.

[0212] Based on the aforementioned pixel matching point pairs, calculate the transformation matrix that maps the field of view of the first image to the field of view of the second image;

[0213] Based on the above transformation matrix, the first image is processed to obtain the third image.

[0214] In one possible implementation, the processing module 702 described above is specifically used for:

[0215] Based on the field of view and resolution information of the second image, the transformation matrix is ​​reversed and mapped to determine the target area of ​​the first image under the field of view of the second image.

[0216] The image outside the target area in the first image is cropped to obtain the third image.

[0217] In one possible implementation, combining Figure 8 ,like Figure 9 As shown, the first region is the stray light region where the stray light is located in the second image; the above-mentioned device 700 further includes: an acquisition module 703; the acquisition module 703 is used to acquire stray light feature information of the stray light in the second image before the processing module 702 performs fusion processing on the first region in the second image and the third image; the processing module 702 is also used to determine the stray light region in the second image based on the stray light feature information and the image features of the second image.

[0218] In one possible implementation, combining Figure 8 ,like Figure 10 As shown, the first region is the stray light region in the second image where the stray light is located; the device 700 further includes: a detection module 704; the detection module 704 is used to detect the brightness value of each pixel in the third image and the second image before the processing module 702 performs fusion processing on the first region in the second image and the third image; the processing module 702 is also used to determine the region where the first pixel is located as the stray light region in the second image;

[0219] The first pixel is a pixel whose brightness value in the second image and the brightness value in the third image are greater than a first preset threshold.

[0220] In one possible implementation, combining Figure 8 ,like Figure 10As shown, the device 700 further includes: a detection module 704; the detection module 704 is used to detect the brightness value of each pixel in each pixel block in the second image before the processing module 702 performs fusion processing on the first region in the second image and the third image to obtain the target image, wherein a pixel block includes at least two pixels; the processing module 702 is also used to determine the region where the first pixel block is located as the stray light region in the second image; wherein the first pixel block is at least one of the following: a pixel block in the second image whose maximum brightness value is greater than a second preset threshold; a pixel block in the second image whose average brightness value is greater than a third preset threshold and whose brightness variance of adjacent pixel blocks is greater than a fourth preset threshold.

[0221] In the image processing apparatus provided in this application embodiment, a first image is first acquired through a first camera, and a second image is acquired through a second camera, wherein the field of view of the first camera includes the field of view of the second camera; then, based on the image features of the first image and the image features of the second image, the first image is processed to obtain a third image; the image content of the third image is matched with that of the second image, and the field of view of the third image is matched with that of the second image; finally, the first region in the second image is fused with the third image to obtain the target image. In this solution, by utilizing the overlapping area of ​​the dual cameras of the electronic device, the first region in the second image that may contain stray light is replaced by an image region in the first image with the same field of view, thereby improving the suppression effect of stray light in imaging.

[0222] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0223] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0224] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0225] Optionally, such as Figure 11 As shown, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0226] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0227] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0228] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0229] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0230] The processor 110 is configured to acquire a first image via a first camera and a second image via a second camera, wherein the field of view of the first camera includes the field of view of the second camera; the processor 110 is further configured to process the first image based on the image features of the first image acquired by the processor 110 and the image features of the second image acquired by the processor 110, to obtain a third image; wherein the third image matches the image content of the second image and the field of view of the third image matches the field of view of the second image; the processor 110 is further configured to fuse a first region in the second image with the third image to obtain a target image.

[0231] In one possible implementation, the processor 110 described above is specifically used for:

[0232] Obtain the first image feature information of the first image and the second image feature information of the second image;

[0233] The first image feature information and the second image feature information are matched to obtain pixel matching point pairs. The pixel matching point pairs include the pixels in the first image and the pixels in the second image that are at the same position as the first pixel.

[0234] Based on the aforementioned pixel matching point pairs, calculate the transformation matrix that maps the field of view of the first image to the field of view of the second image;

[0235] Based on the above transformation matrix, the first image is processed to obtain the third image.

[0236] In one possible implementation, the processor 110 described above is specifically used for:

[0237] Based on the field of view and resolution information of the second image, the transformation matrix is ​​reversed and mapped to determine the target area of ​​the first image under the field of view of the second image.

[0238] The image outside the target area in the first image is cropped to obtain the third image.

[0239] In one possible implementation, the first region is the stray light region in the second image where the stray light is located; the processor 110 is further configured to obtain stray light feature information in the second image before fusing the first region in the second image with the third image to obtain the target image; the processor 110 is further configured to determine the stray light region in the second image based on the stray light feature information and the image features of the second image.

[0240] In one possible implementation, the first region is the stray light region in the second image where the stray light is located; the processor 110 is further configured to detect the brightness value of each pixel in the third image and the second image before performing a fusion process between the first region in the second image and the third image to obtain a target image; the processor 110 is further configured to determine the region where the first pixel is located as the stray light region in the second image.

[0241] The first pixel is a pixel whose brightness value in the second image and the brightness value in the third image are greater than a first preset threshold.

[0242] In one possible implementation, the processor 110 is further configured to, before fusing the first region in the second image with the third image to obtain the target image, detect the brightness value of each pixel in each pixel block in the second image, wherein a pixel block includes at least two pixels; the processor 110 is further configured to determine the region where the first pixel block is located as the stray light region in the second image; wherein the first pixel block is at least one of the following: a pixel block in the second image whose maximum brightness value is greater than a second preset threshold; a pixel block in the second image whose average brightness value is greater than a third preset threshold and whose brightness variance between adjacent pixel blocks is greater than a fourth preset threshold.

[0243] In the electronic device provided in this application embodiment, a first image is first acquired through a first camera, and a second image is acquired through a second camera, wherein the field of view of the first camera includes the field of view of the second camera; then, based on the image features of the first image and the image features of the second image, the first image is processed to obtain a third image; the image content of the third image is matched with that of the second image, and the field of view of the third image is matched with that of the second image; finally, the first region in the second image is fused with the third image to obtain the target image. In this solution, by utilizing the overlapping area of ​​the dual cameras of the electronic device, the first region in the second image that may contain stray light is replaced by an image region in the first image with the same field of view, thereby improving the suppression effect of stray light in imaging.

[0244] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0245] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0246] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0247] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0248] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0249] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0250] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0251] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0252] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0253] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0254] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0256] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: A first image is captured by a first camera, and a second image is captured by a second camera, wherein the field of view of the first camera includes the field of view of the second camera; Based on the image features of the first image and the image features of the second image, the first image is processed to obtain a third image; wherein, the third image matches the image content of the second image and the field of view of the third image matches the field of view of the second image. The first region in the second image is fused with the third image to obtain the target image.

2. The method according to claim 1, characterized in that, The process of processing the first image based on the image features of the first image and the image features of the second image to obtain the third image includes: Obtain the first image feature information of the first image and the second image feature information of the second image; The first image feature information and the second image feature information are matched to obtain pixel matching point pairs. The pixel matching point pairs include pixels in the first image and pixels in the second image that are at the same position as the first pixel. Based on the pixel matching point pairs, calculate the transformation matrix that maps the field of view of the first image to the field of view of the second image; Based on the transformation matrix, the first image is processed to obtain the third image.

3. The method according to claim 2, characterized in that, The process of processing the first image based on the transformation matrix to obtain the third image includes: Based on the field of view and resolution information of the second image, the transformation matrix is ​​reversed and mapped to determine the target area of ​​the first image under the field of view of the second image. The third image is obtained by cropping the image outside the target area in the first image.

4. The method according to claim 1, characterized in that, The first region is the stray light region in the second image where the stray light is located; Before fusing the stray light region in the second image with the third image to obtain the target image, the method further includes: Obtain stray light feature information from the second image; The stray light region is determined based on the stray light feature information and the image features of the second image.

5. The method according to claim 1, characterized in that, The first region is the stray light region in the second image where the stray light is located; Before fusing the stray light region in the second image with the third image to obtain the target image, the method further includes: Detect the brightness value of each pixel in the third image and the second image; The area where the first pixel is located is defined as the stray light area; The first pixel is a pixel whose brightness value in the second image and the brightness value in the third image are greater than a first preset threshold.

6. An image processing apparatus, characterized in that, The device includes: a data acquisition module and a processing module; The acquisition module is used to acquire a first image through a first camera and a second image through a second camera, wherein the field of view of the first camera includes the field of view of the second camera; The processing module is used to process the first image based on the image features of the first image acquired by the acquisition module and the image features of the second image acquired by the acquisition module to obtain a third image; wherein the third image matches the image content of the second image and the field of view of the third image matches the field of view of the second image. The processing module is further configured to fuse the first region in the second image acquired by the acquisition module with the third image to obtain a target image.

7. The apparatus according to claim 6, characterized in that, The processing module is specifically used for: Obtain the first image feature information of the first image and the second image feature information of the second image; The first image feature information and the second image feature information are matched to obtain pixel matching point pairs. The pixel matching point pairs include pixels in the first image and pixels in the second image that are at the same position as the first pixel. Based on the pixel matching point pairs, calculate the transformation matrix that maps the field of view of the first image to the field of view of the second image; Based on the transformation matrix, the first image is processed to obtain the third image.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used for: Based on the field of view and resolution information of the second image, the transformation matrix is ​​reversed and mapped to determine the target area of ​​the first image under the field of view of the second image. The third image is obtained by cropping the image outside the target area in the first image.

9. The apparatus according to claim 6, characterized in that, The device further includes: an acquisition module; the first region is the stray light region where the stray light is located in the second image; The acquisition module is used to acquire stray light feature information in the second image before the processing module fuses the stray light region in the second image with the third image to obtain the target image; The processing module is further configured to determine the stray light region in the second image based on the stray light feature information and the image features of the second image.

10. The apparatus according to claim 6, characterized in that, The device further includes: a detection module; The first region is the stray light region in the second image where the stray light is located; The detection module is used to detect the brightness value of each pixel in the third image and the second image before the processing module fuses the stray light region in the second image with the third image to obtain the target image; The processing module is further configured to determine the area where the first pixel is located as the stray light area in the second image; The first pixel is a pixel whose brightness value in the second image and the brightness value in the third image are greater than a first preset threshold.

Images