Image processing method, electronic device, computer program product and storage medium

Abstract

This application provides an image processing method, an electronic device, a computer program product, and a storage medium, relating to the field of terminal technology. The electronic device includes a first camera and an image sensor, where the first camera can be a main camera lens. In response to a photo-taking operation, the electronic device performs two image output operations based on a first zoom level. The image sensor outputs a first image at a first-size output in full-size mode and a second image at a second-size output in a merged-size mode. The first image has higher resolution than the second image. The image sensor of the electronic device outputs two images of different sizes and generates a fourth image based on these two images, serving as the final third image obtained in response to the user's photo-taking operation. Thus, the electronic device does not need to be equipped with both a main camera lens and a telephoto lens; it can acquire an image that simultaneously includes a wider background and a clearer image using only a single camera and a matching image sensor.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device, computer program product, and storage medium. Background Technology

[0002] Smartphones are equipped with cameras, and their photography capabilities are constantly being optimized. Cameras offer multiple zoom levels, allowing phones to capture images at different zoom ratios. A higher zoom ratio, meaning a longer focal length, results in a smaller field of view, a smaller framing area, and fewer subjects within the frame. Conversely, a lower zoom ratio, meaning a shorter focal length, results in a wider field of view, a larger framing area, and more subjects within the frame.

[0003] A phone's camera captures images at a relatively small zoom level. The wider the field of view, the broader the background is included, but the image sharpness is relatively lower. To obtain a sharper image, the phone's camera switches to a telephoto lens. A telephoto lens has a longer focal length, capturing a sharper image, but with a smaller background.

[0004] The existing image processing solution involves capturing a frame with a large field of view using the main camera lens, and then capturing an image corresponding to the central area of ​​the field of view using the telephoto lens. These two frames are then fused to obtain a single frame with a large field of view and a clear central area.

[0005] Existing image processing solutions rely on electronic devices equipped with both a main camera and a telephoto lens to capture images. When the electronic device does not have a telephoto lens, it is impossible to capture images with a large field of view and clear images in some areas. Summary of the Invention

[0006] This application provides an image processing method, an electronic device, a computer program product, and a storage medium to solve the technical problem that electronic devices without telephoto lenses cannot capture images with a large field of view and clear images in some areas.

[0007] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0008] Firstly, an image processing method is provided, which is applied to an electronic device. The electronic device includes a first camera and an image sensor, which are matched to acquire images. For example, the first camera may include a main camera lens, and the image sensor may include an image sensor matched with the main camera lens. The main camera lens and the image sensor support the electronic device to achieve multi-zoom functionality. The first camera captures light within its field of view, which is then converged to the image sensor, where the image sensor performs photoelectric conversion to generate an image.

[0009] The pixel size of an image sensor is the first size, while the pixel size of the image displayed by an electronic device is the second size, with the first size being larger than the second size. For example, the pixel size of a commonly used main camera lens image sensor is 9000*12000, or 108M, while the pixel size of the displayed image is 3000*4000, or 12M.

[0010] Image sensors primarily output images in two ways: full-size output and merged-size output. Existing electronic devices employ different output methods to send images to the image processor at different zoom levels. Full-size output refers to the image sensor directly reading the charge of a single pixel to obtain corresponding pixel data, ensuring that the pixel specifications of the output image perfectly correspond to the pixel specifications. At a fixed target zoom level, the image sensor outputs an image of the first size to the image processor in full-size output mode. For example, a 108M image sensor supports a target zoom level of 3x for full-size output. Image sensors with different pixel specifications may support different target zoom levels for full-size output.

[0011] Image sensor image merging at a smaller size means that the image sensor combines the charges of adjacent pixels into a single pixel data point. This process of merging charges from all pixels is repeated multiple times before the image is read out, resulting in the final image. Images output using this merged-size-based method have a smaller file size due to the smaller number of pixels. Since this method essentially downsamples the full-size image, the resulting image has relatively lower sharpness compared to the full-size output.

[0012] The existing image capture solution works as follows: if the user selects the target zoom level, the image sensor outputs a full-size image to the image processor. The image processor then crops the center pixel area of ​​the sharp first-size image to obtain a sharp second-size image. If the user selects a zoom level other than the target zoom level, the image sensor outputs a merged second-size image, which is relatively blurred. Regardless of the zoom level, the image sensor outputs only one frame to the image processor.

[0013] The image processing method provided in this application allows an electronic device to reuse its full-size image output capability at multiple different zoom levels, performing two image output operations at one zoom level to acquire two frames of images of different sizes.

[0014] Specifically, the electronic device can display a shooting preview interface, which may include a preview image and the user-selected first zoom level. The electronic device has a camera application installed, which runs various shooting modes, such as photo, portrait, panorama, professional, and video shooting modes. The shooting preview interface can be a preview interface for any of these shooting modes.

[0015] If the electronic device does not receive a click operation on the zoom control, the default zoom level is 1x, meaning the first zoom level is 1x. If the electronic device receives a click operation on the zoom control, it confirms the first zoom level selected by the user based on that click operation.

[0016] If the electronic device receives a photo-taking operation from the user while updating and displaying the preview image on the shooting preview interface, the electronic device will respond to the photo-taking operation by performing image acquisition and processing operations.

[0017] When an electronic device responds to a photo-taking operation, it acquires an image based on a first zoom level. During the process of the image sensor reading the electrical signals converted from each pixel and obtaining pixel data for image output, it performs two image output operations, and the two image output operations output images of different sizes.

[0018] The first image output operation performed by the image sensor is to output a first image at a first size in full-size output mode. The second image output operation performed by the image sensor is to output a second image at a second size in a merged-size output mode. In other words, when the electronic device acquires an image at a first zoom level, unlike existing solutions that output one frame of image to the image processor at one zoom level, this embodiment outputs two frames of images at different sizes to the image processor using two different output methods. The first image has higher resolution than the second image.

[0019] The electronic device's image sensor outputs two frames of images of different sizes. The first image is then cropped using a center-cropping method to obtain a third image. The second and third images are then merged to obtain a fourth image, which serves as the fourth image corresponding to the first zoom level, obtained in response to the user's photo-taking operation.

[0020] The image processing method provided in this application, when an electronic device acquires an image at a first zoom level, reuses the image sensor's function of outputting a full-size image at the target zoom level. The electronic device controls the image sensor to output a first image of the first size in a full-size output manner, and outputs a second image of the second size in a merged-size output manner. The image processor of the electronic device obtains a new fourth image based on the first and second images, which serves as the image corresponding to the current first zoom level. The resulting image can include both a broad background and relatively clear partial pixel areas. The electronic device can obtain high-quality images using only a first camera and a matching image processor. The electronic device does not need to be equipped with both a main camera lens and a telephoto lens simultaneously; it can acquire images that simultaneously include a broader background and a clearer image using only the main camera lens.

[0021] In one possible implementation of the first aspect, the scheme for the electronic device to fuse a second and a third image to generate a fourth image is further specified. Images acquired by the electronic device at different zoom levels correspond to different framing ranges; generally, a higher zoom level results in a larger framing range. When performing image fusion, the electronic device can use the image with a relatively larger framing range as a reference frame and fuse information from the other frame into the reference frame, thereby resulting in a fused image with a larger framing range.

[0022] Specifically, this implementation further defines the relationship between the image's field of view size when the first zoom ratio is less than the target zoom ratio. The field of view of the third image is consistent with the field of view of the image corresponding to the target zoom ratio. When the first zoom ratio is greater than the target zoom ratio, the field of view of the second image is smaller than the field of view of the third image, and the field of view of the third image is consistent with the field of view of the fourth image.

[0023] In one possible implementation of the first aspect, the step of the electronic device fusing the second and third images to generate a fourth image of a second size is further defined. The electronic device can determine whether the first zoom ratio is greater than or less than the target zoom ratio, or receive indication information that the first zoom ratio is greater than the target zoom ratio through other means. If the electronic device determines that the first zoom ratio is greater than the target zoom ratio, or if the electronic device directly receives indication information that the first zoom ratio is greater than the target zoom ratio, and the field of view of the second image obtained based on the first zoom ratio is smaller than the field of view of the third image, the electronic device can use the third image as a reference frame to fuse the second image into the third image to obtain the fourth image. In this way, the fourth image acquired by the electronic device has the same field of view as the third image, has a wider background, and incorporates the higher resolution information of the third image and the higher signal-to-noise ratio information of the second image, resulting in higher image quality.

[0024] In one possible implementation of the first aspect, when the first zoom ratio is less than the target zoom ratio, the size relationship of the image's field of view is further defined. The field of view of the third image is consistent with the field of view of the image corresponding to the target zoom ratio. Since the first zoom ratio is less than the target zoom ratio, the field of view of the second image obtained based on the first zoom ratio is greater than the field of view of the third image. The electronic device fuses the second and third images to obtain a fourth image, and the field of view of the fourth image is consistent with the field of view of the second image, which has the larger field of view.

[0025] In one possible implementation of the first aspect, the step of the electronic device fusing the second and third images to generate a fourth image of a second size is further defined. The electronic device can determine whether the first zoom ratio is less than or greater than the target zoom ratio, or receive indication information that the first zoom ratio is less than the target zoom ratio through other means. If the electronic device determines that the first zoom ratio is less than the target zoom ratio, or if the electronic device directly receives indication information that the first zoom ratio is less than the target zoom ratio, and the field of view of the second image obtained based on the first zoom ratio is greater than the field of view of the third image, the electronic device can use the second image as a reference frame to fuse the third image into the second image to obtain the fourth image. In this way, the fourth image acquired by the electronic device has the same field of view as the second image, has a wider background, and incorporates the higher resolution information of the third image and the higher signal-to-noise ratio information of the second image, resulting in higher image quality.

[0026] In one possible implementation of the first aspect, based on existing processes of the existing electronic device, the ability to obtain an image based on a full-size output method can be performed at the target zoom level. The electronic device can adopt a scheme of acquiring two frames of different sizes and fusing them only when the first zoom level is not equal to the target zoom level.

[0027] Specifically, the step of the electronic device acquiring a first image of a first size and a second image of a second size in response to a photo-taking operation on the shooting preview interface may further include the electronic device acquiring the first image and the second image in response to the photo-taking operation when the first zoom ratio is not equal to the target zoom ratio.

[0028] This preserves the original image output and image processing methods of electronic devices at the target zoom level.

[0029] In one possible implementation of the first aspect, the electronic device acquires a first image at a first zoom level equal to the target zoom level, crops the first image using a center-cropping method, and generates a fifth image of a second size.

[0030] The image sensor of the electronic device has the ability to output a full-size image at the target zoom level. In order to save computing resources, the electronic device can distinguish between the case where the first zoom level is the target zoom level and the case where the first zoom level is not the target zoom level, and perform different image processing operations accordingly.

[0031] In one possible implementation of the first aspect, the step of the electronic device acquiring the second image is further defined. During the process of acquiring a first image of a first size and a second image of a second size in response to a photo-taking operation on the photo-taking preview interface, the electronic device may acquire one frame of the first image and multiple consecutive frames of the second image in response to the photo-taking operation. In the step of fusing the second image and the third image to generate a fourth image of a second size, the electronic device may fuse the third image and multiple consecutive frames of the second image to generate the fourth image of the second size.

[0032] Electronic devices acquire multiple consecutive frames of second images. These frames generally have a consistent field of view, and the information in each frame may not be entirely identical. This information can be complementary to obtain more image information. The electronic device then fuses these multiple frames with a third image, or it can first fuse the multiple frames to obtain a single second image with more information, and then fuse this fused second image with the third image. The resulting fourth image also possesses more image information and has higher image quality.

[0033] In one possible implementation of the first aspect, a step of super-resolution processing of the second image by an electronic device is added. Specifically, before the step of fusing the second and third images to generate a fourth image of a second size, the electronic device may also perform super-resolution processing on the second image. Super-resolution processing of the second image by the electronic device allows the second image to possess more detailed features, improving the clarity of the second image, and thus improving the clarity of the fourth image obtained by fusing the second and third images.

[0034] In other cases, electronic devices can also optimize the first or third image. For example, they can perform noise reduction on the first or third image to improve its signal-to-noise ratio (SNR), thereby improving the SNR of the fourth image obtained by fusing the second and third images. Specifically, the electronic device can first crop the first image to obtain the third image using a center-cropping method, and then perform noise reduction on the third image. This reduces the computational load on the electronic device and improves image processing efficiency.

[0035] In one possible implementation of the first aspect, the image sensor has a pixel size of 108M, the electronic device supports the display of images with a pixel size of 12M, and the target zoom ratio is 3x.

[0036] In one possible implementation of the first aspect, the first size is 9000*12000 and the second size is 3000*4000.

[0037] In one possible implementation of the first aspect, the image sensor has a pixel size of 50M, the electronic device supports the display of images with a pixel size of 12.5M, and the target zoom ratio is 2x.

[0038] Alternatively, in other cases, the image sensor has a pixel size of 200M, the electronic device supports displaying images with a pixel size of 12.5M, and the target zoom ratio is 4x. The data for the first and second dimensions are adjusted accordingly for different image sensor pixel sizes.

[0039] In one possible implementation of the first aspect, the shooting environment during the corresponding photo-taking operation of the electronic device can be further limited to whether it meets the first preset condition. When the electronic device is not in night scene shooting mode, not in high dynamic range shooting mode, or when the real-time brightness value of the first camera's field of view is greater than a brightness threshold, the image sensor outputs the first image in full-size output mode, resulting in a relatively high image signal-to-noise ratio and image quality. The fourth image obtained based on the first image also has high image quality. If the electronic device determines that the current shooting environment meets the first preset condition, it can use the scheme provided in the first aspect of multiplexing full-size output to obtain the first image. The electronic device acquires the first image and the second image of the first size at the first zoom level, crops the first image using a center-cropping method to obtain the third image, and merges the third image and the second image to obtain the fourth image.

[0040] In one possible implementation of the first aspect, if the current shooting environment does not meet the first preset condition, the electronic device may not reuse the full-size image output method. In response to a photo-taking operation on the shooting preview interface, the electronic device obtains a second image of a second size based on the optical image at a first zoom ratio using a merged-size output method. The optical image is a first-size image formed on the image sensor based on the field of view of the first camera in response to the photo-taking operation. The electronic device generates a sixth image of the second size based on the second image. When the current shooting environment does not meet the first preset condition, the electronic device does not acquire the first image with a low signal-to-noise ratio, but instead directly uses the second image to generate the corresponding sixth image, which has a high signal-to-noise ratio.

[0041] In practice, when the camera application is turned on, the electronic device can call the ambient light sensor to collect the ambient light brightness of the shooting environment, determine whether the ambient light brightness meets the first preset condition, and select the corresponding image processing method.

[0042] In a second aspect, this application provides an electronic device including a first camera, an image sensor, a display screen, a memory, and a processor. The first camera, the image sensor, the display screen, and the memory are all coupled to the processor. The memory stores computer-executable instructions. The processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform an image processing method as described in any of the first aspects.

[0043] Thirdly, an electronic device is provided, which has the function of implementing the image processing method of the first aspect described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0044] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, enable the computer to perform the image processing method of any of the first aspects described above.

[0045] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, enables the computer to perform the image processing method of any one of the first aspects described above.

[0046] The technical effects of any of the design methods in aspects two through five can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0047] Figure 1 A schematic diagram of a camera preview interface on an electronic device;

[0048] Figure 2 A schematic diagram of the pixel arrangement and full-size output of an image sensor for an electronic device;

[0049] Figure 3 A schematic diagram for combining the dimensions of an image sensor in an electronic device;

[0050] Figure 4 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0051] Figure 5 A schematic diagram of the interface and images involved in the image processing method provided in the embodiments of this application;

[0052] Figure 6 This is a schematic diagram of the image processing method provided in the embodiments of this application, involving the merging of sizes at 2x magnification;

[0053] Figure 7 A comparative schematic diagram of the field of view of images with different zoom ratios involved in the image processing method provided in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram illustrating the operation flow of the image processing method provided in the embodiments of this application;

[0055] Figure 9 This is a comparative diagram showing images obtained using existing technology and the image processing method provided in this embodiment at non-3x magnification.

[0056] Figure 10 A software framework and flowchart of an image processing method provided in an embodiment of this application;

[0057] Figure 11This is a hardware schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0059] To facilitate understanding, some technical common sense involved in the embodiments of this application will be introduced first.

[0060] Electronic devices are equipped with cameras that can receive user commands to take photos and control the camera to execute the corresponding photo-taking process. The camera of an electronic device mainly includes a shutter, lens module, image sensor, camera driver, and image processing unit (GPU), working together to achieve the photo-taking function. The lens module includes multiple lens groups, which are combined in different ways to create lenses with different focal lengths. When the shutter opens, light enters the lens module, and the lenses converge the light onto the image sensor. The image sensor converts the collected light signal into an electrical signal, which is then transmitted to the image processor for processing, generating an image for display. The camera driver is used to drive the lens module to adjust the focal length and to drive the image sensor to transmit the generated electrical signal to the image processor.

[0061] The image processor mentioned here can be a standalone chip used to perform image processing operations, also known as a display core, visual processor, or display chip. It is a microprocessor specifically designed for performing image and graphics-related calculations in personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). Alternatively, an image processor can also be a functional module within the central processing unit (CPU) or other processors of an electronic device that performs image processing operations; there is no limitation on this. An image processor can call algorithms or neural networks stored within the electronic device to perform image processing operations.

[0062] Electronic devices' lens modules can provide lenses with different focal lengths (or focal ranges) to capture images, achieving zoom functionality. Electronic devices can adjust zoom parameters such as the zoom ratio to change the lens's field of view (FOV), displaying the magnification or reduction of the image's field of view within the shooting interface. The zoom ratios for image capture by electronic devices typically include integer zoom ratios such as 1x, 2x, 3x, and 6x, and may also include non-integer zoom ratios such as 1.5x and 2.9x. Different electronic devices may support different zoom ratios, but commonly include 1x, 2x, and 3x.

[0063] like Figure 1 The diagram shows a schematic of the shooting preview interface. The shooting preview interface mainly includes: a parameter control area 101, an image display area 102, a mode control area 103, a front / rear camera flip control 104, a gallery control 105, a shutter control 106, and a zoom control 107. The parameter control area 101 includes multiple parameter controls, each used to respond to user-input shooting parameter adjustments. The parameter controls included in the parameter control area may include, but are not limited to: flash controls, AI recognition switch controls, color standard controls, and more detailed camera setting controls. The image display area 102 can be used to display a preview image, which is an image captured in real-time by the electronic device through the camera. The electronic device can refresh the display content in the image display area 102 in real-time so that the user can preview the image currently captured by the camera. The mode control area 103 may include multiple mode controls corresponding to different shooting modes, such as photo mode controls, portrait mode controls, video mode controls, professional mode controls, aperture mode controls, night scene mode controls, and more mode controls. Each mode control can be marked with text information, such as "Aperture", "Night Scene", "Portrait", "Photo", "Video", "Pro", "More", or it can be displayed with icons, or a combination of text information and icons.

[0064] The zoom control 107 is used to respond to zoom operations performed by the user on the electronic device. The electronic device responds to these zoom operations by adjusting the field of view of the lens module or the framing range of the generated image. The framing range of the image displayed in the image display area 102 changes with the zoom ratio. For example... Figure 1As shown, the zoom control 107 displays "1x", indicating that the current zoom level of the electronic device is 1x, with "x" used to represent the zoom level thereafter. Users can apply zoom operations by clicking or sliding on the zoom control 107 to adjust the zoom level. Zoom operations can also be applied through touch operations that do not involve the zoom control 107. For example, the electronic device can pre-define sliding fingers towards or away from each other on the shooting preview interface as zoom operations, or pre-define clockwise and counter-clockwise rotation sliding on the shooting interface as zoom operations. These other pre-defined touch operations apply to the shooting interface and not necessarily to the zoom control 107.

[0065] Of course, in other cases, the zoom operation received by the electronic device can also be an adjustment operation on a physical device on the surface or side of the electronic device, rather than an operation on the shooting preview interface. For example, a physical button or knob can be set on the side of the electronic device as a zoom switch. This physical button or knob can be associated with indicating specific zoom value selection or zoom magnification adjustment operations. This physical button or knob can be a separate physical device dedicated to receiving zoom operations, or it can reuse existing volume adjustment buttons or channel adjustment knobs in the shooting scenario, without limitation. This situation may be more suitable for scenarios where the electronic device is a retro-style mobile phone, a feature phone, a point-and-shoot camera, etc. In some other cases, the electronic device can also receive voice control operations from the user to adjust the zoom magnification.

[0066] After receiving a zoom operation from the user, the electronic device can respond to the zoom operation and determine the first zoom magnification indicated by the zoom operation. On one hand, the electronic device can display the first zoom magnification indicated by the zoom operation on the shooting preview interface. On the other hand, the image sensor of the electronic device outputs the original image corresponding to the first zoom magnification to the image processor. The image processor generates an image corresponding to the first zoom magnification based on the original image provided by the image sensor and sends it for display. The following will explain in detail the principle by which the electronic device acquires images at different zoom magnifications through the image sensor and image processor.

[0067] Image sensors utilize the photoelectric conversion function of optoelectronic devices to convert a light image on a photosensitive surface into an electrical signal proportional to the light image. Image sensors are mainly classified into Charge-Coupled Devices (CCDs) and Complementary Metal-Oxide-Semiconductors (CMOSs) based on their device type. A CCD is a semiconductor chip used to capture images, reading pixel data through charge transfer, and has high sensitivity and signal-to-noise ratio. A CMOS is a large-scale integrated circuit chip that reads pixel data through line-by-line scanning. Image sensors in electronic devices can be either CCD or CMOS sensors.

[0068] An image sensor comprises multiple pixels arranged in an array. These pixels divide the overall light image, converged onto the light-receiving surface of the image sensor via a lens, into unit light images. Each pixel then converts its corresponding unit light image into a unit image, which in turn corresponds to each pixel in the image. In other words, in the full-size image output at the original size of the image sensor, there is a one-to-one correspondence between pixels and units.

[0069] like Figure 2 As shown in (1), this is a schematic diagram of the pixel arrangement of an image sensor assembled in an electronic device. In this example, the pixel arrangement size of the image sensor can be 9000*12000, 9000*12000=108000000, and the pixel specification can be recorded as 108M (M stands for Millon). For simplicity, Figure 2 In (1), every 1000 pixels is simplified to one cell. Electronic devices are equipped with image sensors with a 108M pixel specification, and the pixel specification of the generated image is usually 12M, that is, the pixel arrangement size is 3000*4000. It should be noted that the pixel specification and pixel specification mentioned here are usually the specifications and output specifications of the image sensor matched with the main camera lens of the electronic device.

[0070] Based on the pixel specifications of the image sensor and the pixel specifications of the generated image, electronic devices can mainly categorize the image output methods of image sensors into two types: full-size output and binning output. The output method may also differ depending on the zoom level at which the electronic device acquires images.

[0071] The first output method outputs images from the electronic device's image sensor at full size.

[0072] When an electronic device acquires an image at a specific zoom level, the image sensor outputs the image at full size. The zoom level corresponding to when the electronic device outputs the image at full size is denoted as the target zoom level.

[0073] For example Figure 2 As shown in (1), the pixel specification of the image generated by the electronic device is 12M (3000*4000), and the pixel specification of the image sensor is 108M (9000*12000). When the electronic device acquires an image at a 3x zoom ratio, the image sensor outputs the image at full size, that is, the target zoom ratio is 3x zoom ratio.

[0074] like Figure 2 As shown in (2), this is a schematic diagram of an image sensor sensing light to form an optical image, and the electronic device outputs a full-size image to the image processor. For ease of description, the full-size image output by the image sensor is referred to as the first image, and the size of the first image or the size of the image sensor is referred to as the first size.

[0075] In one example, the electronic device supports displaying images with a pixel size of 12M (3000*4000), while the image processor receives a full-size image output from the image sensor with a pixel size of 108M (9000*12000). The image processor needs to crop the first image to achieve a pixel size of 3000*4000. For ease of description, the image obtained after cropping the first image is referred to as the second image, and the size of the second image is called the second size.

[0076] like Figure 2 As shown in (3), the image processor of an electronic device typically uses a center-cropping method. Specifically, the electronic device determines the center pixel of the first image, and uses the center pixel of the first image as the center point to crop the first image with a cropping frame of the second size, i.e., 3000*4000, to obtain the image shown in (3). Figure 2 The second image shown in (4) is used by an electronic device to display the cropped second image as an image captured by a camera at a 3x zoom magnification.

[0077] The image sensor of the electronic device outputs a full-size first image. A second image, directly cropped from this first image without downsampling, is then displayed with higher clarity. (Comparison) Figure 2 The first image shown in (2) and Figure 2 The second image shown in (4) has a larger field of view than the first image, and the clarity of the first image is the same as that of the second image.

[0078] The second method of outputting images involves outputting images of the electronic device's image sensor in a merged size format.

[0079] When an electronic device acquires images at zoom levels other than the target zoom level, the image sensor reads the optical image using a merged readout unit to generate a digital image of the merged size that conforms to the pixel specifications. The generated merged image is then output to the image processor. For ease of description, the zoom levels other than the target zoom level supported by the electronic device are referred to as the second zoom level.

[0080] The optical image reading method of the image sensor merging and reading unit of an electronic device refers to the method of adding the induced charges in adjacent pixels together and reading them out as a single pixel. Binning is divided into horizontal binning and vertical binning. Horizontal binning adds the charges of adjacent rows together, while vertical binning adds the charges of adjacent columns together. Combining several pixels into one pixel improves the sensitivity to light sensing in dark areas and increases the output speed. When electronic devices use binning for both rows and columns simultaneously, the aspect ratio of the image remains unchanged, but the image sharpness decreases. The image merging mode involved in this embodiment can refer to image output using a combined row and column binning mode simultaneously.

[0081] Continuing as mentioned above Figure 2 In the scenario shown in (1), the pixel specification of the image generated by the electronic device is 12M (3000*4000), the pixel specification of the image sensor is 108M (9000*12000), the electronic device supports full-size image output at 3x zoom, and supports merged size image output at other zoom levels besides 3x zoom. The second zoom level that the electronic device supports for merged size image output can include 1x, 2x, 4x, 6x, etc.

[0082] like Figure 3 The image shown is a schematic diagram of an electronic device acquiring an image at a 1x zoom ratio. Figure 3 As shown in (1), this is a full-size optical image generated by the image sensor sensing light. The image sensor merges and reads pixel data. In order to obtain a 3000*4000 image from a 9000*12000 image, the image sensor needs to merge the pixel data of three adjacent pixels and read them out as a single pixel data for output. The merging mentioned here can be understood as three-in-one or nine-in-one. Horizontally, the pixel data of three pixels needs to be merged into one pixel data for reading out, and vertically, the pixel data of three pixels also needs to be merged into one pixel data for reading out.

[0083] like Figure 3As shown in (2), the 9000*12000 pixel array is divided into three parts to obtain pixel regions Z1, Z2...Z11 and Z12. The pixel regions are merged and read to obtain the corresponding pixel regions A1, A2...A11 and A12, as shown in (2). Figure 3 As shown in (3) above, the image obtained after merging and reading pixel data is as follows: Figure 3 As shown in (4) of the diagram. The image sensor at 1x zoom will... Figure 3 The second image shown in (4) is output to the image processor. The image sensor can be read by convolution or downsampling.

[0084] At other zoom levels, the image sensor, based on a 9000*12000 pixel array, adaptively adjusts the number of pixels to be merged and read, so that the final output image has a pixel size of 3000*4000. The specific principle can be referred to the principle of image merging at 1x zoom, which will not be elaborated here.

[0085] contrast Figure 3 The first image shown in (2) is similar to Figure 3 The second image shown in (4) has a first size larger than the second size of the first image, the first image and the second image have the same field of view, and the first image has a higher resolution than the second image.

[0086] Electronic devices can capture a second image at 1x zoom or 3x zoom using their main camera lens. (Comparison) Figure 2 The second image with 3x zoom shown in (4) and Figure 3 The second image shown in (4) has a 1x zoom ratio. The field of view of the image corresponding to the 1x zoom ratio is greater than that of the image corresponding to the 3x zoom ratio. The size of the images is the second size, and the sharpness of the image corresponding to the 1x zoom ratio is less than that of the image corresponding to the 3x zoom ratio.

[0087] In other words, when an electronic device acquires images at different zoom levels, the larger the zoom level, the smaller the field of view of the image, and the image sharpness at other zoom levels is less than that at a 3x zoom level. In reality, when an electronic device captures images using its main camera, the field of view remains constant; the field of view of the generated image shrinks as the zoom level increases. If the electronic device switches from its main camera to a scene-oriented lens, the field of view decreases; conversely, if the electronic device switches from its main camera to a wide-angle lens, the field of view increases.

[0088] When electronic devices capture images, the lens's field of view typically includes foreground and background objects. Foreground objects can be understood as those relatively close to the lens, while background objects can be understood as those relatively far away. Foreground objects have a shallower depth of field, while background objects have a greater depth of field. Foreground objects can include human faces, animal faces, and plant flowers, while background objects can include landscapes and buildings. Relatively speaking, the area occupied by foreground objects within the lens's field of view is smaller than that of background objects. Generally, foreground objects with a shallower depth of field have higher pixel sharpness in the corresponding area of ​​the image, while background objects with a greater depth of field have lower pixel sharpness. A 3x zoom magnification corresponds to high image sharpness but a smaller field of view, including fewer background objects. A 1x zoom magnification corresponds to a larger field of view, including more background objects, but with lower sharpness.

[0089] To capture images that simultaneously include a wider background and a sharper foreground object, electronic devices employ a combination of a main camera and a telephoto lens. Specifically, the main camera and the scene lens each capture a frame within the same field of view. The main camera's field of view is larger than that of the telephoto lens; the first image captured by the main camera has a wider field of view than the second image captured by the telephoto lens. The first image includes more background objects, while the second image has higher sharpness in its central pixel area. The electronic device then merges the higher-sharp pixel area from the second image into the central pixel area of ​​the first image to obtain a new image. This new image includes a wider background and a sharper central pixel area. This image processing scheme relies on the electronic device being equipped with both a main camera and a telephoto lens. Electronic devices without a telephoto lens cannot obtain high-quality images that simultaneously include a wider background and a sharper foreground.

[0090] Based on this, this embodiment provides an image processing method applied to an electronic device. The electronic device may include a first camera and an image sensor. The image sensor can sense the light accumulated after the first camera opens its aperture gate to form an optical image, and convert the optical image into a corresponding digital image. The pixel arrangement specification of the image sensor is denoted as the first size, and the pixel arrangement specification of the image supported for display by the electronic device is denoted as the second size. The image sensor of the electronic device supports full-size image output at a target zoom ratio, i.e., the target zoom ratio is a preset parameter associated with the image sensor. Different image sensors may be associated with different target zoom ratios.

[0091] When an electronic device acquires an image at a target zoom level, it can control the image sensor to output the image at full size, resulting in a high-resolution image. When the electronic device acquires an image at a first zoom level (which is not equal to the target zoom level), it can also reuse the image sensor's full-size output function at the target zoom level. It can control the image sensor to output the first image at full size and the second image by merging sizes, then crop the first image to obtain a third image, and finally merge the second and third images to obtain a fourth image, which serves as the image corresponding to the current first zoom level. In this way, without needing to simultaneously equip a main camera and a scene camera, the image acquired by a single first camera can include both a broad background and a relatively clear central pixel area, resulting in high-quality images. In specific implementations, the first camera can be a main camera supporting multiple zoom levels, or other cameras supporting multiple zoom levels; there are no limitations.

[0092] The image processing method provided in this embodiment is applied to electronic devices, which may include personal computers (PCs), tablet computers, laptops, portable computers (such as mobile phones), wearable electronic devices (such as smartwatches), augmented reality (AR) / virtual reality (VR) devices, in-vehicle computers, and other electronic devices equipped with cameras. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0093] like Figure 4 The diagram shown is a flowchart of an image processing method provided in this embodiment. The provided image processing method mainly includes the following steps:

[0094] S41: The electronic device displays a shooting preview interface; the shooting preview interface includes a first zoom level and a preview image captured by the first camera.

[0095] The image processing method provided in this embodiment is applied to an electronic device, which has an application program installed that can execute the provided image processing method.

[0096] In one example, the electronic device has a separate third-party image processing application installed to perform the provided image processing operations. The electronic device displays an icon for the image processing application on its desktop. If the electronic device receives a click on the icon of the image processing application, it begins to execute the image processing operation corresponding to the provided image processing method.

[0097] In another example, the electronic device has a camera application installed, which runs various shooting modes, such as still, portrait, panorama, professional, and video shooting modes. When the camera application is running in one or more shooting modes, such as still mode, the electronic device begins to execute the provided image processing method. The following will use the still mode within the camera application, with the first camera as the main lens, as an example to explain the specific implementation process of the image processing method provided in this embodiment.

[0098] For example Figure 5 As shown in (1), after the electronic device opens the camera application, it displays a shooting preview interface. The current shooting mode is photo mode (e.g., ...). Figure 5 (1) A). In photo mode, the camera application uses the main camera lens, which works with the image sensor to capture images within the main camera's field of view. The main camera then updates and displays the preview images within the main camera's field of view in real time as a preview stream within the shooting preview interface (e.g., ...). Figure 5 (1) shows B).

[0099] Figure 5 As shown in (1), the shooting preview interface displayed by the electronic device also includes zoom controls (such as...). Figure 5 (1) C). If the electronic device does not receive a click operation on the zoom control, the default zoom ratio is 1x, that is, the first zoom ratio is 1x. If the electronic device receives a click operation on the zoom control, the electronic device confirms the first zoom ratio selected by the user based on the click operation on the zoom control.

[0100] As explained in the foregoing background, the image output by the electronic device's image sensor is related to the first zoom level selected by the user. During the process of updating and displaying the preview image in the preview interface as a preview stream, if the field of view of the first camera changes, or if the first zoom level changes, the field of view of the preview image displayed on the preview interface will also change accordingly. For example, if the first zoom level increases, the field of view of the preview image will decrease; conversely, if the first zoom level decreases, the field of view of the preview image will increase.

[0101] S42: In response to a photo-taking operation on the shooting preview interface, the electronic device acquires a first image of a first size and a second image of a second size; wherein the first image is obtained by full-size output based on an optical image, and the second image is obtained by merged-size output based on an optical image at a first zoom ratio; the optical image is an image of a first size formed on an image sensor based on the field of view of a first camera in response to the photo-taking operation; the clarity of the first image is higher than the clarity of the second image.

[0102] If an electronic device receives a photo-taking action from the user on the photo-taking preview interface while updating and displaying the preview image, the electronic device will respond to the photo-taking action by performing image acquisition and processing operations.

[0103] It should be noted that the photo-taking operation responded to by the electronic device on the shooting preview interface can be a photo-taking operation directly applied to the shooting preview interface, or a photo-taking operation received by the electronic device after the shooting preview interface is displayed. For example, the photo-taking operation can be a click operation of the photo-taking control on the shooting preview interface captured by the electronic device, or it can be a voice control operation containing keywords such as "take a photo" captured by the electronic device, etc., without limitation.

[0104] During the process of updating the shooting preview interface before receiving a photo capture operation, the electronic device may receive multiple zoom operations applied by the user, and the zoom ratio indicated by different zoom operations may be different. The electronic device uses the zoom ratio last determined before receiving the photo capture operation as the first zoom ratio and performs the image processing operation corresponding to the first zoom ratio.

[0105] like Figure 5 As shown in (2), the electronic device responds to the user's action on the camera control (such as...). Figure 5 The click operation shown in (2) (D) opens the shutter. Light within the field of view of the main camera lens passes through the open shutter and converges onto the light-receiving surface of the image sensor via the main camera lens. Each pixel of the image sensor senses the light signal and converts it into a corresponding electrical signal.

[0106] In one example, the first size of the pixel arrangement specification within the image sensor of the electronic device is 9000*12000, the second size of the pixel arrangement specification of the image supported by the electronic device is 3000*4000, and the image sensor of the electronic device supports full-size image output at the target zoom ratio, i.e., 3x magnification.

[0107] In this embodiment, the electronic device acquires images based on a first zoom level. During the process of the image sensor reading the electrical signals converted from each pixel and obtaining pixel data for image output, it performs two output operations. The two output operations output images of different sizes using different output methods.

[0108] In one specific implementation, the first output operation performed by the image sensor is to output a first image of a first size in a full-size output mode.

[0109] like Figure 5As shown in (3), the image sensor reuses the full-size output function at 3x magnification at the first zoom level, and outputs the first image of the first size in the full-size output mode at the first zoom level.

[0110] In another specific implementation, the second output operation performed by the image sensor is to output a second image of a second size in a merged size output manner.

[0111] like Figure 5 As shown in (3), the image sensor retains the operation of merging size output at the first zoom level, reads pixel data at the merged size at the first zoom level, and obtains a second image of the second size.

[0112] Unlike existing solutions where an image sensor outputs one frame of image corresponding to a zoom level to an image processor, the image processing solution provided in this embodiment allows the image sensor to output two frames of different sizes to the image processor in two different output methods when the electronic device acquires an image at the first zoom level.

[0113] The electronic device performs the first output operation, outputting a first image of the first size in a full-size output manner, as described above. Figure 2 The specific schemes for the images and text descriptions shown in (2) and (3) are not repeated here.

[0114] In a specific example, the electronic device acquires a first zoom ratio of 1x, and the image sensor reads pixel data at a merged size to obtain a second image. A specific implementation scheme is as follows: Figure 3 As shown. Figure 3 As shown in (1), this is a full-size 9000*12000 image obtained by the image sensor conversion, as shown in (1). Figure 3 As shown in (2), the 9000*12000 pixel array is divided into three parts to obtain pixel regions Z1, Z2...Z11 and Z12. The pixel regions are merged and read to obtain the corresponding pixel regions A1, A2...A11 and A12, as shown in (2). Figure 3 As shown in (3) above, the second image obtained after merging and reading pixel data is as follows: Figure 3 As shown in (4) above. The image sensor at 1x magnification obtains... Figure 3 The second image shown in (4) is described above. For details on how the image sensor combines images to output a second image at a 1x magnification, please refer to the aforementioned... Figure 3 The proposed solution will not be elaborated further.

[0115] In another specific example, the electronic device acquires a first zoom ratio of 2x, and the image sensor merges size data to read pixel data to obtain a second image. A specific implementation scheme for this can be as follows: Figure 6 As shown. Figure 6 As shown in (1), this is the full-size image obtained by the image sensor, which merges and reads pixel data. The image sensor first merges and reads pixel data in a three-in-one manner, that is, from the full-size image P1 of 9000*12000, as shown in (1). Figure 6 As shown in (2) above, the merged dimensions result in an image P2 with dimensions of 3000*4000. Figure 6 As shown in (3), image P2 is magnified by 2 times to obtain image P3, as follows. Figure 6 As shown in (4), image P3 is then cropped from the central region to obtain image P4. The image P4 obtained after sequentially merging, reading, magnifying, and cropping is the second image output by the image sensor at 2x magnification.

[0116] The above example, using an image sensor with a pixel size of 108M, an image size of 12M, and a target zoom ratio of 3x supporting full-size image output, explains the specific implementation scheme of an electronic device controlling an image sensor to acquire the first and second images. In this case, the image sensor outputs a 108M first image and a 12M second image corresponding to the first zoom ratio. At 1x zoom, the image sensor obtains the second image using a three-in-one or nine-in-one image combining method.

[0117] In other cases, the pixel specifications of the image sensor and the image specifications may also differ.

[0118] For example, in one scenario, if the image sensor has a pixel size of 50M and the image size is 12.5M, then the image sensor supports a target zoom ratio of 2x for full-size image output. At 1x zoom, the camera outputs a 50M first image in full-size output mode, and the image sensor obtains a 12.5M second image by combining two or four images into a single size.

[0119] In another scenario, the image sensor has a pixel size of 200M and an image size of 12.5M. The image sensor supports a target zoom ratio of 4x for full-size output. At 1x zoom, the camera outputs a 200M first image in full-size output mode, and the image sensor outputs a 12.5M second image using a 4-in-1 or 16-in-1 pixel binning method.

[0120] Typically, the pixel size and image size of an image sensor within an electronic device are fixed, as is the target zoom ratio that supports full-size image output. Different electronic devices may have the same or different pixel size, image size, and target zoom ratio. In practice, each electronic device can adaptively adjust the execution of corresponding steps based on the pixel size and image size of its image sensor, without limitation.

[0121] S43: The electronic device fuses the second image and the third image to generate a fourth image of the second size; wherein the third image is a second-sized image obtained by cropping the first image using a center-cropping method; the clarity of the third image is the same as that of the first image, and the clarity of the fourth image is higher than that of the second image.

[0122] like Figure 5 As shown in (3), the electronic device acquires two images of different sizes: a first image of a first size and a second image of a second size. The electronic device first crops the first image using a center-cropping method to obtain a third image, the size of which is also the second size. The electronic device then merges the third image of the second size with the second image of the second size to obtain a fourth image of the second size. The zoom ratio corresponding to the fourth image generated by the electronic device is the first zoom ratio.

[0123] Specifically, the electronic device includes an image sensor and an image processor. The image sensor outputs two frames of images of different sizes to the image processor. The image processor first crops the first image to obtain a third frame, and then generates a fourth frame based on the third and second images, which serves as the final fourth image obtained in response to the user's photo-taking operation.

[0124] In one example, after obtaining the fourth image, the electronic device can directly display the generated fourth image on the interface of the camera app or a third-party app. The electronic device can also display a save control on this interface, allowing it to save the fourth image to the photo album in response to a user's click on the save control. Alternatively, the electronic device can display a cancel control on this interface, allowing it to delete the generated fourth image in response to a user's click on the cancel control.

[0125] In another example, after obtaining the fourth image, the electronic device can directly save the generated fourth image to the photo album. Furthermore, the electronic device can continue displaying the shooting preview interface and update the thumbnail of the previously displayed image in the photo album display area within the shooting preview interface to the thumbnail of the fourth image. If the electronic device receives a click operation on the photo album display area within the shooting preview interface, the electronic device can switch the interface to display the fourth image.

[0126] Electronic devices fuse second and third images to obtain a fourth image. This mainly involves combining the information from the second and third images so that the newly generated fourth image includes the key information and features of both the second and third images. Electronic devices employ image fusion technology to effectively fuse image information from different sources, improving image quality and information content, resulting in a newly generated image with relatively higher clarity and a lower signal-to-noise ratio.

[0127] Image fusion technologies used in electronic devices can include pixel-level fusion, feature-level fusion, and model-level fusion. Pixel-level fusion involves processing and combining pixels from multiple frames one by one, typically using methods such as weighted averaging, maximum or minimum values. Feature-level fusion extracts and matches features from multiple frames, fusing them based on the matching results; this method better preserves image details and features. Model-level fusion models and optimizes information from multiple images to obtain the optimal fusion result. It can employ models such as wavelet transform, multi-scale analysis, and deep learning for image fusion, and can better handle multiple frames of images at different scales and resolutions.

[0128] In this embodiment, the electronic device fuses the second image and the third image. The second image is obtained through a merged output method, resulting in relatively low resolution but a relatively high signal-to-noise ratio. The third image is obtained by center-cropping the first image obtained from full-size output, resulting in relatively high resolution but a relatively low signal-to-noise ratio.

[0129] Besides key parameters like sharpness and signal-to-noise ratio, another crucial image parameter is the field of view. For the same main camera lens on the same electronic device, the field of view varies depending on the zoom level. For example... Figure 7 The image shown is a comparison diagram of the field of view for images at different zoom levels. Figure 7 As shown in (1), the 3x image has a larger field of view than the 5x image, and the 3x image has a smaller field of view than the 2x image. Figure 7 As shown in (2), for the same main camera lens of the same electronic device, the field of view of the same lens gradually decreases as the zoom ratio increases. It should be noted that... Figure 7 (2) in the figure only illustrates the size of the image's field of view and does not take into account the image output size. When generating an image based on the zoom ratio, electronic devices also perform operations such as merging pixels, enlarging, and cropping on the image so that the size of the output image is always the second size.

[0130] When fusing a second and a third image, an electronic device needs to select one frame as a reference frame and fuse the information from the other frame into it. This ensures that the fourth image has characteristics such as a relatively large field of view, relatively high clarity, and a relatively high signal-to-noise ratio. Therefore, when performing image fusion, the electronic device selects an image with a relatively large field of view as the reference frame to ensure that the fused image retains a large field of view.

[0131] Continue as Figure 4 As shown, after executing S42', acquiring a first image of a first size and a second image of a second size, and cropping the first image using a center-cropping method to obtain a third image of the second size, the electronic device can also perform the following steps:

[0132] S42A: The electronic device determines whether the first zoom ratio is greater than the target zoom ratio.

[0133] If the first zoom ratio is greater than the target zoom ratio, the electronic device executes S431: the electronic device fuses the second image into the third image to obtain the fourth image.

[0134] The electronic device acquires a second and a third image, where the field of view of the third image corresponds to the field of view of the image at the target zoom level. If the first zoom level determined by the electronic device is greater than the target zoom level, and the field of view of the second image is smaller than that of the third image, the third image with a relatively larger field of view can be used as a reference frame.

[0135] The electronic device performs image fusion, fusing information from the second image into the third image to obtain a fourth image. In this way, the field of view of the fourth image is the same as that of the third image, larger than that of the second image, and it incorporates the high signal-to-noise ratio information of the second image and the high-resolution information of the third image.

[0136] If the first zoom ratio is less than the target zoom ratio, the electronic device executes S432: the electronic device merges the third image into the second image to obtain the fourth image.

[0137] The electronic device determines a first zoom ratio that is less than the target zoom ratio. The second image's field of view is larger than the third image's field of view; therefore, the second image, with its relatively larger field of view, can be used as a reference frame. The electronic device performs image fusion, integrating information from the third image into the second image to obtain a fourth image. Thus, the fourth image's field of view is consistent with that of the second image, larger than that of the third image, and incorporates the high signal-to-noise ratio information of the second image and the high-resolution information of the third image.

[0138] In a specific example, such as Figure 8The diagram illustrates the process of an electronic device performing image processing. The camera APK collects information and transmits it to the image sensor via a service host, where the camera outputs an image. The image sensor senses light to form an optical image, outputting a first image in full-size mode. A third image is obtained by cropping the first image using a center-cropping method. The first size is 9000*12000, the second size is 3000*4000, and the target zoom ratio is 3x. Based on the optical image, a merged-size output method is used, followed by switching processing to obtain the second image, which can output multiple consecutive frames of the second image. The image processor can perform single-frame noise reduction on the third image, resulting in a noise-reduced third image, denoted as P1. The image processor can also perform super-resolution processing on multiple consecutive frames of the second image, resulting in a super-resolution second image, denoted as P2.

[0139] The electronic device determines whether the first zoom level is greater than the target zoom level. If the first zoom level is greater than the target zoom level, P1 and P2 are fused together using P1 as the reference frame to obtain the fourth image. If the first zoom level is less than the target zoom level, P1 and P2 are fused together using P2 as the reference frame to obtain the fourth image.

[0140] The image processing method provided in this embodiment can select an image as a reference frame in image fusion based on the relationship between the first zoom ratio and the target zoom ratio, so that the fused fourth image has a larger field of view, i.e. a wider background, and features such as higher signal-to-noise ratio and sharpness, resulting in higher image quality.

[0141] In one specific implementation, the electronic device can select different execution schemes based on the relationship between the first zoom ratio and the target zoom ratio. Continuing as... Figure 4 As shown, after the electronic device executes S41 and displays the shooting preview interface, in response to the shooting operation, it can also perform the following steps:

[0142] S41A: Electronic device determines whether the first zoom level is equal to the target zoom level.

[0143] The image sensor of the electronic device has the ability to output a full-size image at the target zoom level. In order to save computing resources, the electronic device can distinguish between the case where the first zoom level is the target zoom level and the case where the first zoom level is not the target zoom level, and perform different image processing operations accordingly.

[0144] If the first zoom ratio is not equal to the target zoom ratio, then execute S42', S42A, S431 and S432, that is, acquire the first image of the first size and the second image of the second size respectively, crop the first image to obtain the third image by centering, and select the frame with the relatively larger field of view from the second image and the third image as the reference frame according to the size of the first zoom ratio and the target zoom ratio, so as to merge the second image and the third image to obtain the fourth image, so that the fourth image has a larger field of view, higher clarity and higher signal-to-noise ratio.

[0145] If the first zoom ratio is equal to the target zoom ratio, execute S441: The electronic device responds to the photo-taking operation and acquires a first image of a first size.

[0146] S442: The electronic device uses a center-cropping method to crop the first image and generate a fifth image of the second size.

[0147] The first zoom ratio is the target zoom ratio. For example, for a 108MP image sensor, the first zoom ratio is 3x. In this case, based on its full-size image output capability, the image sensor directly outputs a full-size first image to the image processor at 3x zoom. The image processor then crops the first image, for example... Figure 2 As shown in (3), the fifth image is obtained by cropping the central pixel area. Since the full-size image has high clarity, the cropped fifth image also has high clarity. That is, a high-quality image with high clarity can be obtained without acquiring and fusing multiple frames. When an electronic device acquires the fifth image, it can directly send the fifth image to the display or save it, or it can perform noise reduction and other optimization and quality improvement processing on the fifth image before sending it to the display or saving it.

[0148] In this embodiment, the electronic device distinguishes between cases where the first zoom level is the target zoom level and cases where the first zoom level is not the target zoom level. This not only saves computing resources but also obtains high-quality images.

[0149] like Figure 9 The image shown is a comparative diagram of images obtained using existing technology and the image processing method provided in this embodiment at a non-3x magnification. Figure 9 As shown in (1), this is an image acquired by an existing image acquisition method at a 2x zoom ratio. Figure 9 As shown in (2), this is an image obtained by the image processing method provided in this embodiment at a 2x zoom ratio. It can be clearly determined that the image obtained by the image processing method provided in this embodiment has improved details and higher image clarity. Multiple simulation data prove that the image processing method provided in this embodiment can improve the clarity of digital zoom without telephoto devices to a certain extent.

[0150] The electronic device executes step S43, which involves fusing the second and third images to obtain the fourth image. There are multiple possible implementation schemes for this step.

[0151] In one example, the electronic device can use an image fusion algorithm to fuse image information from a second image with image information from a third image to obtain a fourth image. The image fusion algorithm used by the electronic device can be any commonly used image fusion algorithm and is not limited to any particular one.

[0152] In another example, the electronic device can also load a pre-trained image processing model, which can then be used to process the second and third images to obtain a fourth image.

[0153] The model is essentially a trained neural network. A neural network is a network formed by connecting multiple individual neural units together. The output of one neural unit can be the input of one or more other neural units. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0154] Neural networks used to train image processing models typically include deep neural networks (DNNs) or convolutional neural networks (CNNs). Deep neural networks, also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of the layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the multiple layers in between are hidden layers. Layers can be fully connected; that is, any neuron in the i-th layer can be connected to any neuron in the (i+1)-th layer.

[0155] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers; this feature extractor can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to a subset of neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each of which can consist of a series of rectangularly arranged neural units.

[0156] Electronic devices typically require multiple sets of sample images for model training. Each set includes two types of images: input sample images containing features to be classified, and target sample images containing target features. It's important to note that the features to be classified refer to those requiring processing such as recognition, classification, and labeling, such as features indicating a clear or blurred face. Target features are the specific types or categories of features that need to be identified or processed, such as features indicating image fusion or clarity. The image content of the input and target sample images is correlated.

[0157] In this system, the input sample image serves as the input value to the neural network, while the target sample image serves as the target value. The input sample image contains various basic elements that the image to be processed by the image processing model may include. The target sample image, on the other hand, can be an image obtained by processing the input sample image according to user requirements. This target sample image may be obtained by removing elements that the user does not need from the input sample image. In other words, the elemental content of the input sample image and the target sample image is basically the same, but their feature attributes may differ.

[0158] In this embodiment, the electronic device can train a neural network to obtain an image processing model, and then use the image processing model to perform image processing operations. Alternatively, the electronic device can directly obtain image processing models trained by other devices and use them to execute the image processing method provided in this embodiment.

[0159] The process of training a neural network to obtain an image processing model by an electronic device mainly includes: the electronic device acquires a sample image and a neural network without setting relevant parameters of the weight matrix; the sample image is input into the neural network for iterative training; the weight matrix in the neural network is gradually optimized according to the loss function until the loss function converges, so that the weight matrix in the neural network reaches the optimal value; at this point, the neural network can be used as an image processing model.

[0160] The sample images acquired by the electronic device include input sample images and target sample images. The input sample images may include a first sample image and a second sample image, both having a second pixel size. The first sample image includes a wider background and sample feature objects; the pixel regions containing these feature objects in the first sample image have lower sharpness and a higher signal-to-noise ratio. The second sample image also includes sample feature objects; the pixel regions containing these feature objects in the second sample image have higher sharpness and a lower signal-to-noise ratio. The electronic device may also label the first sample image as a reference frame, enabling the trained image processing model to retain a larger field of view based on the reference frame.

[0161] The target sample image also has a second pixel size. The target feature object includes the sample feature object and a wider background. The pixel region where the sample feature object is located in the target sample image has higher clarity. Compared to the first sample image, the pixel region where the sample feature object is located in the target sample image has higher clarity.

[0162] The electronic device uses the first and second sample images as input values ​​and the target sample training as the target value to iteratively train a neural network without weight parameters. Specifically, the electronic device uses a loss function to calculate the difference between the predicted value and the true value, and then uses the difference for backpropagation to modify the weight matrix; then, the training sample data is input into the neural network after the weight matrix is ​​modified, and multiple iterations of training are performed until the difference is minimized, that is, the neural network converges.

[0163] When a neural network iterates through input and target values, it employs a backpropagation algorithm to correct the parameters of the initial weight matrix during training, minimizing the reconstruction error loss. Specifically, the difference between the predicted and target values ​​obtained in each iteration is calculated and then propagated back into the neural network to modify the weight matrix. The modified neural network is then iterated through again, and the difference from each iteration is used to update the weight matrix. This process of iterative training and weight matrix modification continues until the difference between the predicted and target values ​​falls within the acceptable loss range of the loss function, indicating convergence. This is how neural network convergence is achieved through multiple iterative training iterations. The backpropagation algorithm, primarily driven by error loss, is used to obtain the optimal parameters of the neural network model. Of course, other similar algorithms can also be used for iterative optimization, without limitation.

[0164] The electronic device trains an image processing model based on the above scheme and stores the trained image processing model. When the image processor performs fusion processing on the second and third images, it can input the second and third images into the image processing model for image processing, and select either the second or third image as a reference frame based on the magnitude of the first zoom ratio and the target zoom ratio, so that the image processing model outputs a new image with the same field of view as the reference frame, which is then used as the fourth image.

[0165] In other implementations, such as Figure 8As shown, the electronic device can also detect whether the current shooting environment meets a preset first condition through preview scene detection. Based on this, this embodiment can also limit the prerequisite for performing image processing operations to a first preset condition. The first preset condition may include at least one of the following: the electronic device is not in night scene shooting mode, the electronic device is not in high dynamic range (HDR) shooting mode, and the real-time brightness value of the first camera's field of view is greater than a brightness threshold. If the shooting environment of the electronic device meets the first preset condition, the signal-to-noise ratio of the first image obtained by the electronic device through the image sensor in full-size output mode is relatively high, resulting in a relatively high signal-to-noise ratio of the fused fourth image, which is suitable for reusing the full-size output mode. Conversely, if the shooting environment does not meet the first preset condition, the signal-to-noise ratio of the second image obtained by the electronic device through the image sensor in full-size output mode is low, and the signal-to-noise ratio of the fused image is also relatively low, making it unsuitable for reusing the full-size output mode. Specifically, after receiving a photo-taking operation, the electronic device can also first detect whether the current shooting environment meets the first preset condition.

[0166] If the current shooting environment meets the first preset conditions, the electronic device responds to the photo-taking operation, acquires the first image and the second image, crops the first image using a center cropping method to obtain the third image, and merges the second image and the third image to obtain the fourth image.

[0167] The shooting environment meets the first preset condition. The electronic device, at the first zoom level, reuses the full-size output method available at the target zoom level, outputting a first image of the first size and a second image of the second size. The first image is then cropped using a center-cropping method to obtain a third image. The second and third images are then merged to obtain a fourth image. This results in a fourth image with a high signal-to-noise ratio and high clarity.

[0168] If the current shooting environment does not meet the first preset conditions, the electronic device responds to the photo-taking operation, acquires a second image of a second size, and generates a sixth image of a second size based on the second image.

[0169] In low-light or night scene shooting environments, the image sensor outputs the first image in full-size mode, resulting in a low signal-to-noise ratio and poor image quality. The third and fourth images obtained based on the poor first image are also of poor quality.

[0170] In this scenario, the electronic device can directly acquire a second image at a second size without using the full-size output method, and generate a sixth image based on the second image. The electronic device can directly display or save the second image as the sixth image, or it can perform noise reduction or super-resolution optimization on the second image before displaying or saving it as the sixth image.

[0171] In another scenario, the electronic device can also utilize the ambient light sensor to collect the ambient light intensity of the shooting environment when the camera application is activated. If the ambient light intensity meets the brightness requirements for a bright environment, the image processing method provided in this embodiment can be executed to obtain a high-quality image. If the ambient light intensity does not meet the brightness requirements for a bright environment, the image processing method provided in this embodiment can be omitted.

[0172] The image processing method provided in the above embodiments can obtain high-quality images that simultaneously include a wider background and a clearer central pixel area based on the main lens of the camera. When the image sensor's first zoom ratio is not the target zoom ratio, it reuses the on-chip zoom capability at the target zoom ratio, expanding the cropped area after full-size output from the central pixel area to the pixel area guided by the user's point of interest. This provides 3X high-definition image quality for the user's area of ​​interest. Through subsequent image fusion capabilities, it allows the user's more focused areas to present higher clarity, achieving zoom fusion based on a single camera and solving the dependence problem of telephoto lenses.

[0173] The above embodiments explain the specific implementation process of the image processing method from the perspective of electronic devices. The following will explain the specific implementation process of the image processing method from the perspective of the internal software architecture of electronic devices.

[0174] like Figure 10 The diagram shown illustrates the internal software architecture and simplified flowchart of an electronic device. The following section will explain in detail the image processing flow of the electronic device, taking into account its internal architecture.

[0175] Specifically, the internal architecture of an electronic device can be divided into four layers, from top to bottom: the application layer (APP), the framework layer (FWK), the hardware abstraction layer (HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included, without limitation.

[0176] The application layer may include a series of application packages, such as the camera application involved in this embodiment. In addition, the application layer also includes applications such as a gallery and image processing applications with camera functionality. Application packages may also include applications such as calling, calendar, maps, navigation, music, video, and text messaging.

[0177] The framework layer provides the application programming interface (API) and programming framework for applications in the application layer. The framework layer includes some predefined functions.

[0178] The framework layer runs a camera service, which can be called by camera applications to implement shooting-related functions. In addition, the framework layer may include a window manager, content providers, a view system, a resource manager, and a notification manager. The window manager manages window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to applications. Data can include video, images, and audio. The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. The display interface can consist of one or more views. The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, notifications can be used to announce download completion or message alerts. Notifications can also appear as icons or scrollbar text in the system's top status bar, such as notifications from background applications, or as dialog windows on the screen. Notifications can include text messages displayed in the status bar, sound alerts, vibrations from the terminal, flashing indicator lights, etc. It should be noted that the camera application can also invoke content providers, resource managers, notification managers, window managers, view systems, etc., according to actual business needs; this embodiment does not impose any restrictions on this.

[0179] The kernel layer is the layer between hardware and software. The kernel layer contains at least a camera driver. This camera driver can be used to drive hardware modules with shooting capabilities, such as image sensors. In other words, the camera driver is responsible for data interaction with the image sensor. The kernel layer may also include display drivers, audio drivers, sensor drivers, etc., but this embodiment does not impose any limitations on this. In this solution, the electronic device includes a camera, the camera includes a main lens, the camera driver is used to drive the main lens to open the shutter and adjust the lens focal length of the lens combination, and the image sensor interacts with the camera driver to realize the camera's image acquisition function.

[0180] The Hardware Abstraction Layer (HAL) can encapsulate drivers in the kernel layer and provide an interface for the framework layer to call them, shielding the underlying hardware implementation details. For example... Figure 10As shown, the aforementioned hardware abstraction layer may include a camera call processing module (Camera HAL), etc. The camera call processing module is the core software framework of the camera, and it includes an interface module, a sensor call module (Sensor Node), an image processing module, etc. These interface module, sensor call module, and image processing module are components in the image data and control command transmission pipeline within the camera call processing module. The image processing module can call the image processor to implement the corresponding image processing functions mentioned above.

[0181] Specifically, the sensor module can be a control node for the image sensor, which can control the image sensor through the camera driver. The interface module can be a software interface for the application framework layer, used for data interaction with the application framework layer. Of course, the interface module can also interact with image processing modules in the camera's call processing module.

[0182] The image processing module can process the first and second images output by the image sensor, crop the first image using a center cropping method to obtain the third image, and fuse the second and third images to obtain the fourth image.

[0183] like Figure 10 As shown, at the application layer, the camera application can receive the user's zoom operation and pass the first zoom level selected by the user to the camera service at the frame layer.

[0184] At the frame layer, the camera service receives the first zoom level, generates a photo request, and the photo request indicates the first zoom level and the way the image sensor outputs the image: outputting a first image of the first size in full-size output mode and / or outputting a second image of the second size in merged-size output mode.

[0185] The camera service stores the target zoom ratio. If it is determined that the first zoom ratio is not the target zoom ratio, the generated photo request instructs the image sensor to output the image in the following ways: outputting a first image of the first size in full-size output mode, and outputting a second image of the second size in merged-size output mode.

[0186] If the camera service determines that the first zoom level is the target zoom level, the generated photo request instructs the image sensor to output the first image in full-size mode.

[0187] The camera service sends photo requests to the hardware abstraction layer via the application programming interface (API).

[0188] In the hardware abstraction layer, the interface module receives the photo-taking request and passes it to the sensor calling module and the image processing module respectively.

[0189] In one scenario, where the first zoom ratio is not the target zoom ratio, the sensor invocation module can invoke the image sensor via the camera driver, causing the image sensor to output a first image of a first size and a second image of a second size. The image sensor outputs the first image of the first size in full-size output mode.

[0190] The image sensor first downsamples to obtain a second-size image by merging sizes, then enlarges the second-size image according to the zoom ratio, and finally crops it according to the center pixel area to obtain a second-size image.

[0191] The image processing module receives the first image, crops the first image using a center-cropping method, and obtains the third image.

[0192] The image processing module receives the second image, merges the second image and the third image to obtain the fourth image.

[0193] The image processing module uploads the fourth image (which can be uploaded to the framework layer via the interface module), where the camera service and view system of the framework layer render the fourth image, and then uploads it to the application layer for display or saving.

[0194] In another scenario, where the first zoom ratio is the target zoom ratio, the sensor invocation module can invoke the image sensor via the camera driver, causing the image sensor to output a first image of the first size in full-size output mode. The image processing module receives the first image and crops it using a center-cropping method to obtain the third image. The image processing module then processes the third image to obtain the fifth image.

[0195] The image processing module uploads the fifth image (which can be uploaded via the interface module) to the framework layer. The camera service and view system in the framework layer render the fifth image, and then upload it to the application layer for display or saving.

[0196] Understandable, Figure 10 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0197] Furthermore, it is understood that the electronic device, in order to implement the image processing method of this embodiment, includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0198] In addition, embodiments of this application also provide an electronic device, including a first camera, an image sensor, a display screen, a memory, and a processor, wherein the first camera, image sensor, display screen, memory, and processor are coupled together;

[0199] The memory stores the instructions that the computer executes;

[0200] The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the image processing method provided in the above embodiments. In addition to these main components, the electronic device also includes components for implementing basic functions, which will be discussed below. Figure 11 Please provide a detailed explanation.

[0201] like Figure 11 The diagram shown is a structural schematic of an electronic device 1100 provided in an embodiment of this application. The electronic device 1100 may include a processor 1110, an external memory interface 1120, an internal memory 1121, a Universal Serial Bus (USB) interface 1130, a charging management module 1140, a power management module 1141, a battery 1142, antenna 1, antenna 2, a mobile communication module 1150, a wireless communication module 1160, an audio module 1170, a speaker, a receiver, a microphone, a headphone jack, a sensor module 1180, buttons 1190, a motor 1191, an indicator 1192, a camera 1193, a display screen 1194, and a SIM card module 1195, etc. The sensor module 1180 may include a pressure sensor 1180A, a gyroscope sensor 1180B, an ambient light sensor 1180C, a touch sensor 1180D, etc.

[0202] The illustrated structure of this embodiment does not constitute a limitation on the electronic device 1100. It may include more or fewer components than illustrated, or combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0203] Processor 1110 may include one or more processing units. For example, processor 1110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image processor 1110A or an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0204] The aforementioned controller can be the decision-maker that directs the various components of the electronic device 1100 to work in a coordinated manner according to instructions. It is the nerve center and command center of the electronic device 1100. The controller generates operation control signals based on the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0205] The processor 1110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1110 is a cache memory, which can store instructions or data that the processor 1110 has just used or that are used repeatedly. If the processor 1110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1110, and thus improves the efficiency of the system.

[0206] In some embodiments, the processor 1110 may include interfaces. These interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0207] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDL) and a serial clock line (SCL). In some embodiments, the processor 1110 may include multiple I2C buses. The processor 1110 can couple to the touch sensor 1180D, charger, flash, camera 1193, etc., through different I2C bus interfaces. For example, the processor 1110 can couple to the touch sensor 1180D through the I2C interface, enabling the processor 1110 and the touch sensor 1180D to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 1100.

[0208] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1110 to a camera 1193, a display screen 1194, a wireless communication module 1160, an audio module 1170, a sensor module 1180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0209] USB port 1130 can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 1130 can be used to connect a charger to charge electronic device 1100, or to transfer data between electronic device 1100 and peripheral devices. It can also be used to connect headphones for audio playback. Furthermore, it can be used to connect other electronic devices, such as AR devices.

[0210] The interface connection relationships between the modules illustrated in this embodiment of the invention are merely illustrative and do not constitute a structural limitation on the electronic device 1100. The electronic device 1100 may employ different interface connection methods or a combination of multiple interface connection methods as described in this embodiment of the invention.

[0211] The charging management module 1140 can be a rechargeable battery or a disposable battery. If it's a rechargeable battery, it can receive charging input via a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1140 can receive charging input from the wired charger via the USB interface 1130. In some wireless charging embodiments, the charging management module 1140 can receive wireless charging input via the wireless charging coil of the electronic device 1100. While charging the battery 1142, the charging management module 1140 can also supply power to the electronic device 1100 via the power management module 1141.

[0212] The power management module 1141 connects the battery 1142, the charging management module 1140, and the processor 1110. The power management module 1141 receives input from the battery 1142 and / or the charging management module 1140, providing power to the processor 1110, memory 1121, external memory interface 1120, display 1194, camera 1193, and wireless communication module 1160, etc. The power management module 1141 can also monitor parameters such as the charging management module capacity, charging management module cycle count, and charging management module health status (leakage current, impedance), etc. In some embodiments, the power management module 1141 may also be located within the processor 1110. In some embodiments, the power management module 1141 and the battery 1142 may also be located in the same device.

[0213] The wireless communication function of electronic device 1100 can be realized through antenna 1, antenna 2, mobile communication module 1150, wireless communication module 1160, modem, and baseband processor.

[0214] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, a cellular antenna can be multiplexed as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in conjunction with a tuning switch.

[0215] The mobile communication module 1150 provides a communication processing module for wireless communication solutions, including 2G / 3G / 4G / 5G, applied to the electronic device 1100. The mobile communication module 1150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 1150 receives electromagnetic waves from the antenna 1, filters and amplifies the received electromagnetic waves, and transmits them to the modem for demodulation. The modem may include a modulator and a demodulator.

[0216] The wireless communication module 1160 provides a communication processing module for solutions of wireless communication used in electronic devices 1100, including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Fodulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 1160 can be one or more devices integrating at least one communication processing module. The wireless communication module 1160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 1110. The wireless communication module 1160 can also receive signals to be transmitted from processor 1110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0217] In some embodiments, antenna 1 of electronic device 1100 is coupled to mobile communication module 1150, and antenna 2 is coupled to wireless communication module 1160, enabling electronic device 1100 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include Global Positioning System (SBAS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation System (SBAS).

[0218] Electronic device 1100 implements display functions through a GPU, a display screen 1194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 1110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0219] Display screen 1194 is used to display images, videos, etc. Display screen 1194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), Minied, MicroLED, Micro-OLED, Quantum Dot Light-Emitting Diodes (QLED), etc. In some embodiments, electronic device 1100 may include one or N displays 1194, where N is a positive integer greater than 1.

[0220] Electronic device 1100 can perform shooting functions through ISP, camera 1193, video codec, GPU, display screen and application processor.

[0221] The ISP (Image Signal Processor) is used to process data fed back from the camera 1193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 1193.

[0222] Camera 1193 is used to capture still images or videos. Camera 1193 may include a camera 1193A and an image sensor 1193B. An object is projected onto the image sensor through the camera to generate an optical image. The image sensor may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 1100 may include one or N cameras 1193, where N is a positive integer greater than 1.

[0223] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when electronic device 1100 selects a frequency point, the DSP is used to perform Fourier transforms on the frequency energy.

[0224] Video codecs are used to compress or decompress digital video. Electronic device 1100 may support one or more video codecs. Thus, electronic device 1100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0225] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0226] The external storage interface 1120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1100. The external memory card communicates with the processor 1110 through the external storage interface 1120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0227] Internal memory 1121 can be used to store computer executable program code, which includes instructions. Processor 1110 executes various functional applications and data processing of electronic device 1100 by running the instructions stored in internal memory 1121. Memory 1121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 1100 (such as audio data, phonebook, etc.). Furthermore, memory 1121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, other volatile solid-state storage devices, Universal Flash Storage (UFS), etc.

[0228] Electronic device 1100 can implement audio functions, such as music playback and recording, through audio module 1170, speaker, receiver, microphone, headphone jack, and application processor.

[0229] The audio module 1170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 1170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 1170 may be located in the processor 1110, or some functional modules of the audio module 1170 may be located in the processor 1110.

[0230] Buttons 1190 include a power button, volume buttons, etc. Buttons 1190 can be mechanical buttons or touch-sensitive buttons. Electronic device 1100 receives input from buttons 1190 and generates key signal inputs related to user settings and function control of electronic device 1100.

[0231] Motor 1191 can generate vibration alerts. Motor 1191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, touch operations applied to different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations applied to different areas of the display screen 1194 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0232] Indicator 1192 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0233] The SIM card module 1195 is used to implement the communication function of the SIM card. The SIM card module 1195 may include a SIM card interface, SIM card circuitry, and related auxiliary devices. The SIM card can be inserted into or removed from the SIM card interface to achieve contact and separation with the electronic device 1100. In some embodiments, the electronic device 1100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 1100 and cannot be separated from it.

[0234] The image processing methods described in the foregoing embodiments can all be implemented in the electronic device 1200 having the above-described hardware structure.

[0235] Based on the above embodiments, this application also provides an image processing apparatus, which includes a processor for executing the image processing method provided in the above embodiments.

[0236] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method provided in the above embodiments.

[0237] This application also provides a computer program product containing instructions that, when run on a computer, enable the computer to perform the image processing method provided in the above embodiments.

[0238] The specific implementation methods and technical effects of the electronic devices, computer-readable storage media, and computer program products containing instructions provided in this application can be found in the specific implementation process and technical effects of the image processing method provided in the foregoing embodiments, which will not be repeated here.

[0239] In some embodiments, as described above, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the functional modules described above is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0240] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0241] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0242] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device including a first camera and an image sensor; the image processing method includes: The shooting preview interface is displayed; wherein, the shooting preview interface includes a first zoom level and a preview image captured by the first camera; In response to a photo-taking operation on the shooting preview interface, a first image of a first size and a second image of a second size are acquired; the first image is obtained based on an optical image in a full-size output mode, and the second image is obtained based on the optical image in a merged-size output mode according to the first zoom ratio; the optical image is an image of the first size formed on the image sensor based on the field of view of the first camera in response to the photo-taking operation; the sharpness of the first image is higher than that of the second image. The second image and the third image are merged to generate a fourth image of the second size; wherein the third image is an image of the second size obtained by cropping the first image using a center-cropping method; the clarity of the third image is the same as that of the first image, and the clarity of the fourth image is higher than that of the second image.

2. The image processing method according to claim 1, characterized in that, The first zoom ratio is greater than the target zoom ratio, the field of view of the second image is smaller than the field of view of the third image, and the field of view of the third image is the same as the field of view of the fourth image; wherein, the target zoom ratio is a preset parameter associated with the image sensor.

3. The image processing method according to claim 2, characterized in that, The step of fusing the second image and the third image to generate a fourth image of the second size includes: The first zoom ratio is greater than the target zoom ratio, and the second image is fused into the third image to obtain the fourth image.

4. The image processing method according to any one of claims 1-3, characterized in that, The first zoom ratio is less than the target zoom ratio, the field of view of the second image is greater than the field of view of the third image, and the field of view of the second image is the same as the field of view of the fourth image; wherein, the target zoom ratio is a preset parameter associated with the image sensor.

5. The image processing method according to claim 4, characterized in that, The step of fusing the second image and the third image to generate a fourth image of the second size includes: The first zoom ratio is less than the target zoom ratio, so the third image is fused into the second image to obtain the fourth image.

6. The image processing method according to any one of claims 2 or 3, characterized in that, The step of acquiring a first image of a first size and a second image of a second size in response to a photo-taking operation on the shooting preview interface includes: In response to the photo-taking operation, if the first zoom level is not equal to the target zoom level, the first image and the second image are acquired.

7. The image processing method according to any one of claims 1-3, characterized in that, The step of acquiring a first image of a first size and a second image of a second size in response to a photo-taking operation on the shooting preview interface includes: In response to the photo-taking operation, a first image frame and multiple consecutive second images are acquired; The step of fusing the second image and the third image to generate a fourth image of the second size includes: The third image and multiple consecutive frames of the second image are fused to generate the fourth image of the second size.

8. The image processing method according to any one of claims 1-3, characterized in that, Before the step of fusing the second image and the third image to generate a fourth image of the second size, the image processing method further includes: Super-resolution processing is performed on the second image.

9. The image processing method according to claim 6, characterized in that, The image processing method further includes: The first zoom ratio is equal to the target zoom ratio, and the first image is acquired; the target zoom ratio is a preset parameter associated with the image sensor; The first image is cropped using a center-cropping method to generate a fifth image of the second size.

10. The image processing method according to any one of claims 1-3, characterized in that, The first camera includes a main camera lens.

11. The image processing method according to any one of claims 1-3, characterized in that, The image sensor has a pixel size of 108M, the electronic device supports displaying images with a pixel size of 12M, and the target zoom ratio is 3x.

12. The image processing method according to claim 11, characterized in that, The first dimension is 9000*12000, and the second dimension is 3000*4000.

13. The image processing method according to any one of claims 1-3, characterized in that, The image sensor has a pixel size of 50M, the electronic device supports displaying images with a pixel size of 12.5M, and the target zoom ratio is 2x. Alternatively, the image sensor has a pixel size of 200M, the electronic device supports displaying images with a pixel size of 12.5M, and the target zoom ratio is 4x.

14. The image processing method according to any one of claims 1-3, characterized in that, The step of acquiring a first image of a first size and a second image of a second size includes: The current shooting environment meets the first preset conditions, and the first image and the second image are acquired; wherein, the first preset conditions include at least one of the following: the electronic device is not in night scene shooting mode, the electronic device is not in high dynamic range shooting mode, and the real-time brightness value of the field of view of the first camera is greater than a brightness threshold.

15. The image processing method according to claim 14, characterized in that, The image processing method further includes: If the current shooting environment does not meet the first preset condition, in response to the shooting preview interface, a second image of a second size is acquired; the second image is obtained based on the optical image according to the first zoom ratio and in a merged size output method; the optical image is the first size image formed on the image sensor based on the field of view of the first camera in response to the shooting operation. A sixth image of the second size is generated based on the second image.

16. An electronic device, characterized in that, The electronic device includes a first camera, an image sensor, a display screen, a memory, and a processor, wherein the first camera, the image sensor, the display screen, and the memory are all coupled to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the image processing method as described in any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the image processing method as described in any one of claims 1 to 15.

18. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 15.

Images