Image processing method, electronic device, computer program product, and storage medium

Abstract

The present application provides an image processing method, an electronic device, a computer program product and a storage medium, which relate to the field of terminal technology. The camera of the electronic device includes a main camera lens and a camera sensor. In response to a photo-taking operation, the electronic device performs two image output operations based on a target zoom ratio. The camera sensor outputs a first image of a first size in a full-size output mode, and outputs a second image of a second size in a merged-size output mode. The clarity of the first image is higher than the clarity of the second image. The camera sensor of the electronic device outputs two frames of images of different sizes to an image processor, and the image processor generates a frame of a third image based on the two frames of images of different sizes as a third image finally obtained in response to the user's photo-taking operation. In this way, the electronic device does not need to be equipped with a telephoto lens, and can obtain an image that includes a wider background and a clearer image only through the main camera lens.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to an image processing method, electronic device, computer program product, and storage medium. Background Art

[0002] Mobile phones are equipped with cameras, and their photography capabilities are becoming increasingly sophisticated. Cameras offer multiple zoom ratios, allowing phones to capture images at different zoom levels. The larger the zoom ratio, or focal length, the smaller the camera's field of view, the smaller the framing area, and the fewer subjects captured within the frame. Conversely, the smaller the zoom ratio, or focal length, the larger the camera's field of view, the larger the framing area, and the more subjects captured within the frame.

[0003] Mobile phone cameras capture images at a smaller zoom ratio. A wider field of view includes a wider background, but the image clarity is relatively lower. To obtain a clearer image, the mobile phone camera switches to a telephoto lens. A telephoto lens has a longer focal length, resulting in a clearer image but a smaller background.

[0004] The existing image processing solution is to use the main camera lens to capture a frame of image with a larger viewing range, and then use the telephoto lens to capture the image corresponding to the central area of ​​the viewing range. The two frames of image are fused to obtain a frame of image with a larger viewing range and a clear central area.

[0005] The images captured by existing image processing solutions rely on the electronic device being equipped with both a main camera lens and a telephoto lens. If the electronic device does not have a telephoto lens, it is impossible to capture images with a large viewing range and clear areas. Summary of the Invention

[0006] The embodiments of the present application provide an image processing method, an electronic device, a computer program product, and a storage medium, which are used to solve the technical problem that an electronic device cannot capture an image with a large viewing range and clear areas when it does not have a telephoto lens.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device includes a camera and an image processor. The camera captures images, and the image processor processes the images. The camera includes a main camera lens and a camera sensor. The main camera lens captures light within a viewing range and converges it onto the camera sensor, which then performs photoelectric conversion to generate an image.

[0009] The pixel size of the camera sensor is a first size, and the pixel size of the image supported by the electronic device for display is a second size, where the first size is larger than the second size. For example, the pixel size of the camera sensor of a common main camera lens is 9000*12000, or 108M, and the pixel size of the displayed image is 3000*4000, or 12M.

[0010] The ways in which camera sensors output images mainly include full-size output and merged-size output. Existing electronic devices use different output methods to output images to image processors at different zoom ratios.

[0011] Full-size image output refers to the camera sensor directly reading the charge of a single pixel to obtain the corresponding pixel data. This ensures that the pixel specifications of the output image fully correspond to the pixel specifications. The camera sensor outputs a first-size image to the image processor at a fixed first zoom factor. For example, a 108M camera sensor supports a first zoom factor of 3x for full-size image output. Camera sensors with different pixel specifications may support different first zoom factors for full-size image output.

[0012] The camera sensor's binned output method combines the charges of adjacent pixels into a single pixel data set. This process is repeated multiple times before being read out to produce the corresponding image. Images output by the camera sensor using binned output contain fewer pixels and are therefore smaller in size. Because binned output is effectively a downsampling of the full-size image, images produced using binned output have lower clarity than those produced using full-size output.

[0013] In existing photography solutions, if the user selects a first zoom ratio, the camera sensor outputs a full-size, clear image of the first size to the image processor. The image processor then crops the central pixel area of ​​the first-size clear image to produce a second-size, clear image. If the user selects a zoom ratio other than the first, the camera sensor outputs a merged, relatively blurred image of the second size. Regardless of the zoom ratio, the camera sensor only outputs a single frame of image to the image processor.

[0014] The image processing method provided in this embodiment enables the electronic device to reuse the full-size image output capability of the electronic device at multiple different zoom factors.

[0015] Specifically, the electronic device may display a shooting preview interface, which may include a preview image and a user-selected target zoom factor. The electronic device may be installed with a camera application that operates multiple shooting modes, such as photo, portrait, panorama, professional, and video. The shooting preview interface may be a shooting preview interface for any of these shooting modes.

[0016] If the electronic device does not receive a click operation on the zoom control, the default zoom ratio is 1x, that is, the target zoom ratio is 1x. If the electronic device receives a click operation on the zoom control, the electronic device confirms the target zoom ratio selected by the user based on the click operation on the zoom control.

[0017] When the electronic device is updating and displaying a preview image on a shooting preview interface, if it receives a photo-taking operation from a user, the electronic device performs image acquisition and processing operations in response to the photo-taking operation.

[0018] In response to a photo-taking operation, the electronic device captures an image based on a target zoom ratio. In the process of reading the electrical signal converted by each pixel and obtaining pixel data for image output, the camera sensor performs two image output operations, and the sizes of the images output by the two image output operations are different.

[0019] The first image output operation performed by the camera sensor is to output a first image of a first size in a full-size output mode. The second image output operation performed by the camera sensor is to output a second image of a second size in a combined-size output mode. In other words, when the electronic device captures an image at a target zoom factor, unlike existing solutions that output a single image frame per zoom factor to an image processor, the camera sensor in this embodiment outputs two images of different sizes to the image processor using two different output modes. The clarity of the first image is higher than that of the second image.

[0020] The camera sensor of the electronic device outputs two images of different sizes to an image processor. The image processor generates a third image based on the two images of different sizes as the final third image obtained in response to a user's photo taking operation. The zoom ratio of the third image generated by the electronic device is the target zoom ratio.

[0021] The image processing method provided in this embodiment is that when an electronic device captures an image at a target zoom ratio, the electronic device reuses the function of the camera sensor to output a full-size image at a first zoom ratio, and controls the camera sensor to output a first image of the first size in full-size output and a second image of the second size in a combined size output. The image processor of the electronic device obtains a new image frame based on the first image and the second image as the image corresponding to the current target zoom ratio. The image obtained in this way can include both a wider background and a clearer partial pixel area. In this way, the electronic device does not need to be equipped with a telephoto lens, and can obtain an image that includes both a wider background and a clearer image using only the main camera lens.

[0022] In a possible implementation manner of the first aspect, a solution for an image processor of the electronic device to fuse the first image and the second image to generate the third image is further limited.

[0023] Specifically, the image processor obtains a first image output by the camera sensor in a full-size output mode. The first image includes a wider background and a clearer foreground feature object or other feature object. This part of the foreground feature object or other feature object may be the feature object that the user focuses on. The electronic device crops the first image to obtain a fourth image. The size of the fourth image cropped by the electronic device is recorded as the third size, and the third size is less than or equal to the second size. Since the fourth image is directly cropped from the first image, the clarity of the first image is consistent with the clarity of the fourth image, and both are higher than the clarity of the second image output by the merged output mode.

[0024] The image processor fuses the fourth image obtained by cropping the first image with the second image to obtain a third image. Thus, the third image includes both the wider background in the second image and the higher-resolution pixel area in the fourth image.

[0025] In one possible implementation of the first aspect, the method for cropping the first image by the image processor to obtain the fourth image is further limited. In this implementation, the pixel region containing the target feature object of interest to the user is used as the region of interest, and the clarity of the region of interest in the acquired third image is maximized. For example, the target feature object can be a human face, an animal face, a flower, or other feature object of interest to the user. Of course, the electronic device can also provide a custom editing control to receive identification information of the target feature object input by the user, thereby improving the clarity of the pixel region containing such target feature object.

[0026] The image processor obtains the real-time coordinate range of a target feature object in the preview image. The target feature object is a predetermined object that is within the framing range of the primary camera. For example, the image processor may obtain the real-time coordinate range of a face in the preview image of the preview stream.

[0027] The image processor maps the real-time coordinate range of the target feature object in the preview image to the pixel area determined in the first image according to the pixel mapping relationship between the preview image and the first image, and determines the first pixel area corresponding to the target feature object in the first image.

[0028] Based on the determined first pixel region of the target feature object, the image processor crops a block containing the first pixel region from the first image to obtain a fourth image. Because the fourth image is directly cropped, it includes the second pixel region corresponding to the target feature object; the first pixel region and the second pixel region have the same clarity.

[0029] In a possible implementation of the first aspect, the step of the image processor cropping a block containing the first pixel area from the first image to obtain the fourth image is further defined. The image processor determines a target cropping frame based on the center pixel point of the first pixel area. The center point of the target cropping frame coincides with the center pixel point of the first pixel area, the size of the target cropping frame is a second size, the pixel area covered by the target cropping frame at least partially overlaps with the first pixel area, or the pixel area covered by the target cropping frame may also completely include the first pixel area. The image processor crops the first image according to the target cropping frame to obtain the fourth image.

[0030] In a possible implementation manner of the first aspect, the clarity of corresponding pixel areas in the images before and after fusion is further limited.

[0031] A definition scheme for the clarity of a pixel region corresponding to a target feature object in an image includes: the second image includes a third pixel region corresponding to the target feature object, and the third image includes a fourth pixel region corresponding to the target feature object. The clarity of the fourth pixel region is higher than the clarity of the third pixel region, and the clarity of the second pixel region is the same as the clarity of the fourth pixel region.

[0032] The definition scheme for the clarity of other pixel areas in the image except the pixel area corresponding to the target feature object includes: the second image also includes a fifth pixel area except the third pixel area, the third image also includes a sixth pixel area except the fourth pixel area, and the clarity of the fifth pixel area is the same as the clarity of the sixth pixel area.

[0033] In a possible implementation manner of the first aspect, the step of the image processor fusing the second image and the fourth image to obtain the third image is further limited.

[0034] The image processor superimposes the pixel features of the second pixel region in the fourth image onto the third pixel region in the second image to obtain a third image. The image processor can call an image fusion algorithm or a trained image processing model to perform image fusion processing.

[0035] In a possible implementation manner of the first aspect, the electronic device is limited to select different execution schemes according to a relationship between the target zoom magnification and the first zoom magnification.

[0036] The electronic device determines whether the target zoom ratio is the first zoom ratio. If the target zoom ratio is the first zoom ratio, the electronic device controls the camera sensor to output a first image of a first size to the image processor in response to the photographing operation, and crops the first image to obtain a fifth image.

[0037] If the target zoom ratio is not the first zoom ratio, the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to the image processor in response to the photographing operation, and generates a third image based on the first image and the second image.

[0038] The camera sensor of the electronic device has the ability to output full-size images at a first zoom ratio. In order to save computing resources, the electronic device can distinguish between the cases where the target zoom ratio is the first zoom ratio and the cases where the target zoom ratio is not the first zoom ratio, and perform different image processing operations respectively.

[0039] In a possible implementation manner of the first aspect, a pixel specification of the camera sensor is 108M, a pixel specification of an image supported for display by the electronic device is 12M, and the first zoom ratio is 3x.

[0040] In a possible implementation of the first aspect, a pixel specification of the camera sensor is 50M, a pixel specification of an image supported for display by the electronic device is 12.5M, and the first zoom ratio is 2x;

[0041] or,

[0042] The pixel specification of the camera sensor is 200M, the pixel specification of the image supported by the electronic device for display is 12.5M, and the first zoom ratio is 4x.

[0043] In one possible implementation of the first aspect, whether the shooting environment in which the electronic device performs the photo-taking operation is a high-light environment may be further limited. When the shooting environment is a high-light environment, the camera sensor outputs the first image in full-size format, and the image signal-to-noise ratio is high and the image quality is good. When the shooting environment is a low-light environment, the camera sensor outputs the first image in full-size format, and the image signal-to-noise ratio is low and the image quality is poor. The quality of the third image obtained based on the poor first image is also poor.

[0044] When the camera application is turned on, the electronic device can use the ambient light sensor to collect the ambient light brightness of the shooting environment. If the ambient light brightness meets the brightness requirements for a high-brightness environment, the image processing method provided in this embodiment can be executed to obtain a high-quality image. If the ambient light brightness does not meet the brightness requirements for a high-brightness environment, the image processing method provided in this embodiment may not be executed.

[0045] The electronic device detects whether the current photography mode is low-light photography mode or night photography mode. If the current photography environment is low-light photography mode or night photography mode, the electronic device controls the camera sensor to output a second image of a second size according to a target zoom ratio to the image processor, and displays the second image. If the target zoom ratio is not low-light photography mode or night photography mode, the electronic device executes an operation of controlling the camera sensor to output a first image of a first size and a second image of a second size, respectively, to the image processor in response to a photography operation.

[0046] In a second aspect, the present application provides an electronic device including a camera, a memory, and a processor, wherein the camera and the memory are coupled to the processor;

[0047] Memory stores computer-executable instructions;

[0048] The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the image processing method as described in any one of the first aspects.

[0049] In a third aspect, an electronic device is provided that has the function of implementing the image processing method of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.

[0050] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer can execute any one of the image processing methods in the first aspect.

[0051] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any one of the image processing methods in the first aspect.

[0052] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a shooting preview interface of an electronic device;

[0054] Figure 2 Schematic diagram of pixel arrangement and full-size output of camera sensor for electronic devices;

[0055] Figure 3 A schematic diagram for plotting the combined dimensions of camera sensors for electronic devices;

[0056] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0057] Figure 5 Schematic diagram of the interface and image involved in the image processing method provided in the embodiment of the present application;

[0058] Figure 6 A schematic diagram of merging and outputting images at a 2x magnification involved in the image processing method provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of cropping a first image to obtain a fourth image according to the image processing method provided in an embodiment of the present application;

[0060] Figure 8 Another schematic diagram of cropping a first image to obtain a fourth image according to the image processing method provided in an embodiment of the present application;

[0061] Figure 9 A schematic diagram of the image processing method provided in an embodiment of the present application;

[0062] Figure 10 A schematic diagram of the processing flow of the image processing method provided in an embodiment of the present application for different target zoom ratios;

[0063] Figure 11 A software framework and flowchart of the image processing method provided in the embodiments of the present application;

[0064] Figure 12 A hardware schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0066] To facilitate understanding, some technical common sense involved in the embodiments of this application is first introduced.

[0067] Electronic devices equipped with cameras can receive user photo-taking commands and control the camera to execute the corresponding photo-taking process. The camera in an electronic device primarily includes a shutter, lens module, camera sensor, camera driver, and image processor, which work together to realize the photo-taking function. The lens module includes multiple lens groups, which are used in different combinations to form lenses with different focal lengths. When the shutter is opened, light enters the lens module, which focuses the light onto the camera sensor. The camera sensor converts the collected light signal into an electrical signal, which is then transmitted to the image processor for processing and generates an image for display. The camera driver is used to drive the lens module to adjust the focal length and the camera sensor to transmit the generated electrical signal to the image processor. The image processor (GPU) mentioned here can be a separate chip used to perform image processing operations. It is also called a display core, visual processor, or display chip. It is a microprocessor specifically designed to perform image and graphics-related operations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). Alternatively, the image processor may be a central processing unit (CPU) of an electronic device or a functional module in other processors that performs image processing operations, without limitation. The image processor may call an algorithm or neural network stored in the electronic device to perform image processing operations.

[0068] The lens module of an electronic device can provide lenses with different focal lengths (or focal lengths) to capture images, which is called zoom. By adjusting the zoom parameters such as the zoom ratio (Zoom Ratio) of the zoom to adjust the field of view (FOV) of the shooting interface, the technical effect of magnifying or reducing the subject in the shooting interface is achieved. The zoom ratios of images captured by electronic devices usually include: 1x, 2x, 3x, 6x, etc. The zoom ratios supported by different electronic devices may be different. Most electronic devices include 1x (1x), 2x (2x), and 3x (3x).

[0069] like Figure 1As shown in (1), it is the shooting preview interface (such as Figure 1 The schematic diagram of U1) shown in (1) in FIG. The shooting preview interface mainly includes: a parameter control area 101, an image display area 102, a mode control area 103, a front and rear camera flip control 104, a gallery control 105, a shutter control 106 and a zoom control 107. Among them, the parameter control area 101 includes multiple parameter controls, each of which is used to respond to the shooting parameter adjustment operation input by the user. The parameter controls included in the parameter control area may include but are not limited to: a flash control, an AI recognition switch control, a color standard control, and a more detailed camera setting control. The image display area 102 can be used to display a preview image, which is an image captured in real time by the electronic device through the camera. The electronic device can refresh the display content in the image display area 102 in real time to facilitate the user to preview the image currently captured by the camera. The mode control area 103 may include multiple mode controls corresponding to different shooting modes, such as a photo mode control, a portrait mode control, a video mode control, a professional mode control, an aperture mode control, a night scene mode control and more mode controls. Each mode control can be marked only by text information, for example, "Aperture", "Night Scene", "Portrait", "Photo", "Video", "Professional", "More", or it can be displayed by an icon, or a combination of text information and icons.

[0070] The zoom control 107 is used to respond to the user's zoom operation on the electronic device to adjust the field of view of the lens module. The image displayed in the image display area 102 changes with the adjustment of the field of view of the lens module. Figure 1 As shown in (1), "1x" is displayed on the zoom control 107, indicating that the current zoom magnification of the electronic device is 1x, and the subsequent x represents the zoom magnification. The user can apply a zoom operation to adjust the zoom magnification by applying a click operation or a sliding operation on the zoom control 107. Of course, the zoom operation can also be applied by a touch operation that does not act on the zoom control 107. For example, the electronic device can pre-define the operation of sliding fingers toward each other or sliding fingers away from each other on the shooting preview interface as a zoom operation, or pre-define the operation of rotating clockwise and rotating counterclockwise on the shooting interface as a zoom operation. Such other pre-defined touch operations act on the shooting interface, but not necessarily on the zoom control 107.

[0071] Of course, in other cases, the zoom operation received by the electronic device may also be a zoom operation that does not need to act on the shooting preview interface, but an adjustment operation on a physical device on the surface or side of the electronic device. For example, a physical button or knob is set on the side of the electronic device as a zoom switch, and this part of the physical buttons or knobs can be associated with instructions for specific zoom value selection or zoom ratio adjustment operations, etc. This part of the physical buttons or knobs can be a separate physical device dedicated to receiving zoom operations, or it can be a reused existing volume adjustment button or channel adjustment knob in the shooting scene, etc., without limitation. This situation may be more suitable for scenarios where the electronic device is a retro-style mobile phone, an elderly phone, a card camera, etc.

[0072] After receiving a user-initiated zoom operation, the electronic device responds to the zoom operation and determines the target zoom factor indicated by the zoom operation. The camera sensor then outputs a raw image corresponding to the target zoom factor to the image processor. The image processor then generates an image corresponding to the target zoom factor based on the raw image provided by the camera sensor and displays the image. The following details the principle by which an electronic device acquires images at different zoom factors using a camera sensor and an image sensor.

[0073] Camera sensors use the photoelectric conversion function of photoelectric devices to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. Camera sensors are mainly divided into charge-coupled devices (CCDs) and complementary metal oxide semiconductors (CMOSs) based on the device type. Among them, CCDs are semiconductor chips used to capture images. They read pixel data through charge transfer and have high sensitivity and signal-to-noise ratio. CMOS is a large-scale integrated circuit chip that reads pixel data through line-by-line scanning. The camera sensor of an electronic device can be a CCD sensor or a CMOS sensor.

[0074] A camera sensor consists of multiple pixels arranged in an array. These pixels divide the overall light image, which is focused onto the sensor's light-receiving surface by a lens, into unit light images. Each pixel then converts its corresponding unit light image into a unit image, with each unit image corresponding to a pixel in the image. This means that in the full-size image output at the camera sensor's native size, there is a one-to-one correspondence between pixels and pixels.

[0075] like Figure 2(1) in FIG. 1 is a schematic diagram of the pixel arrangement of a camera sensor assembled in an electronic device. In this example, the pixel arrangement size of the camera sensor is 9000*12000, 9000*12000=108000000, and the pixel specification can be recorded as 108M (M stands for Millon). For simple illustration, Figure 2 In (1), every 1000 pixels are simplified to 1 cell. The electronic device is equipped with a camera sensor with a 108M pixel size, and the pixel size of the generated image is usually 12M, that is, the pixel arrangement size is 3000*4000. It should be noted that the pixel size and pixel size mentioned here are usually the specifications of the main camera lens of the electronic device's camera and the camera sensor that matches the main camera lens.

[0076] Based on the pixel specifications of the camera sensor and the pixel specifications of the generated image, electronic devices can mainly categorize camera sensor image output methods into two types: full-size output and merged-size output. Electronic devices may also use different output methods when capturing images at different zoom ratios.

[0077] In the first output method, the camera sensor of the electronic device outputs the image at full size.

[0078] When the electronic device captures an image at a certain zoom ratio, the camera sensor outputs the image at full size. The zoom ratio corresponding to the electronic device outputting the image at full size is recorded as the first zoom ratio.

[0079] For example Figure 2 As shown in (1), the pixel size of the image generated by the electronic device is 12M (3000*4000), and the pixel size of the camera sensor is 108M (9000*12000). When the electronic device captures an image at a 3x zoom ratio, the camera sensor outputs the image in full size, that is, the first zoom ratio is a 3x zoom ratio.

[0080] like Figure 2 As shown in (2), the camera sensor outputs a full-size image to the image processor, and the full-size image output by the camera sensor is recorded as the first image.

[0081] The pixel specification of the image supported for display by the electronic device is 12M, i.e. 3000*4000, while the pixel specification of the full-size image output by the camera sensor received by the image processor is 108M, i.e. 9000*12000. The image processor needs to crop the first image so that the pixel specification of the cropped image is 3000*4000, and the cropped image is recorded as the second image.

[0082] like Figure 2As shown in (3), the image processor of the electronic device usually adopts the following cropping scheme: determine the center pixel point of the first image, and crop the first image with a cropping frame of 3000*4000 based on the center pixel point of the first image, to obtain the following Figure 2 The electronic device can display the cropped second image as an image captured by the camera at a 3x magnification.

[0083] The camera sensor of the electronic device outputs a full-size first image, and the second image directly cropped from the full-size first image is not subjected to downsampling processing, so the clarity of the displayed image is higher.

[0084] In the second output method, the camera sensor of the electronic device outputs the image in a combined size.

[0085] When the electronic device captures an image at a zoom factor other than the first zoom factor, the camera sensor reads pixel data from the pixel array using a binning method to generate an image of a binned size that meets the pixel specifications, and outputs the generated binned image to the image processor. For ease of description, among the multiple zoom factors supported by the electronic device, zoom factors other than the first zoom factor are referred to as second zoom factors.

[0086] The way in which the camera sensor of an electronic device merges and reads pixel data means that the charges induced in adjacent pixels are added together and read out in a one-pixel mode. Binging is divided into horizontal Binging and vertical Binging. Horizontal Binging adds the charges of adjacent rows together and reads out, while vertical Binging adds the charges of adjacent columns together and reads out. Combining several pixels to act as one pixel improves the sensitivity to light induction in dark places and the output speed. When Binging is used for rows and columns at the same time, the aspect ratio of the image does not change, and the resolution of the image is reduced. The merged output mode mentioned in this embodiment may refer to the use of the merged output mode for rows and columns at the same time.

[0087] Continue as above Figure 2 In the scenario shown in (1), the pixel size of the image generated by the electronic device is 12M (3000*4000), the pixel size of the camera sensor is 108M (9000*12000), and the electronic device supports full-size image output at a 3x magnification. At zoom magnifications other than 3x, it supports merged-size image output. The second zoom magnifications supported by the electronic device for merged-size image output may include 1x, 2x, 4x, 6x, etc.

[0088] like Figure 3 As shown in FIG, it is a schematic diagram of an electronic device collecting images at a 1x magnification. Figure 3As shown in (1), the full-size image converted by the camera sensor is the pixel data read by the camera sensor. In order to obtain a 3000*4000 image from a 9000*12000 image, the camera sensor needs to merge the pixel data of three adjacent pixels and read them out as one pixel data output. The merging mentioned here can be understood as three-in-one or nine-in-one. The pixel data of three pixels needs to be merged into one pixel data read out horizontally, and the pixel data of three pixels needs to be merged into one pixel data read out vertically.

[0089] like Figure 3 As shown in (2), the 9000*12000 pixel array is divided into three parts to obtain pixel areas Z1, Z2...Z11 and Z12. The pixel areas are merged and read to obtain the corresponding pixel areas A1, A2...A11 and A12, as shown in Figure 3 As shown in (3) in the figure, the image obtained after merging and reading pixel data is as follows Figure 3 As shown in (4) in the figure. The camera sensor is at 1x magnification. Figure 3 The second image shown in (4) is output to the second image processor. The camera sensor can be combined and read in a convolution or downsampling manner.

[0090] At other zoom ratios, the camera sensor adaptively adjusts the number of pixels it reads, based on its 9000 x 12000 pixel array, to achieve a final output image size of 3000 x 4000 pixels. The specific principles for this are similar to those for outputting images at 1x magnification, and will not be elaborated on here.

[0091] contrast Figure 2 and Figure 3 Based on a substantially identical or identical framing range, the full-size image captured by the camera sensor is the same. In response to different zoom magnifications, the camera sensor outputs different images to the image processor in different output modes. The image acquired by the electronic device at the first zoom magnification, i.e., 3x, is directly cropped from the center area of ​​the full-size image, so the image clarity is high and the background range is smaller than the framing range. The image acquired by the electronic device at the second zoom magnification has a background range that is consistent with the framing range, but the image clarity is reduced due to the merged output.

[0092] In many scenarios where electronic devices capture images, it's common for the framing range to include both foreground and background objects. Foreground objects can include human faces, animal faces, or flowers, while background objects can include landscapes and buildings. Relatively speaking, the foreground object's area within the framing range is smaller than the background object's area. Consequently, in the image captured by the electronic device, the area of ​​the first pixel region corresponding to the foreground object is smaller than the area of ​​the second pixel region corresponding to the background object.

[0093] For example Figure 1 As shown in (1), the electronic device captures an image at a common magnification (e.g., 1x), and in the captured image, the first pixel area corresponding to the face object (e.g., Figure 1 Z1 shown in (1) is smaller than the second pixel area corresponding to the background object (such as Figure 1 Z2) shown in (1) in the figure, which can retain more background objects, that is, capture a wider background. Figure 1 As shown in (2), Figure 1 The image obtained by proportionally cropping the first pixel area shown in (1) shows that the clarity of the face object is low, the features such as facial expressions and skin details are less, and the image quality is poor. The image clarity that users actually want is as follows. Figure 1 As shown in (3), the face object has higher clarity and more features such as facial expressions and skin details. Such images can be considered as high-quality images.

[0094] In order to obtain a high-quality image that includes a wider background and a clearer face, the electronic device uses a main camera lens and a telephoto lens to capture a frame of image respectively, and then fuses the two frames of image processing scheme. Specifically, the electronic device calls the main camera lens to capture a first image corresponding to the target zoom ratio indicated by the zoom operation, and the first image includes a wider background. The electronic device calls the telephoto lens to capture a second image, and the second image includes a smaller background, but the clarity of the central pixel area is higher. The electronic device fuses the pixel area with higher clarity in the second image into the central pixel area of ​​the first image to obtain a new image, and the new image includes an image of a wider background and a clearer central pixel area. This image processing scheme relies on the electronic device being equipped with a telephoto lens. For electronic devices that do not have a telephoto lens, it is impossible to obtain a high-quality image that includes a wider background and a clearer foreground.

[0095] Based on this, this embodiment provides an image processing method for use in an electronic device. The camera of the electronic device may include a main camera lens and a camera sensor that matches the main camera lens for imaging. The pixel arrangement specification of the camera sensor is recorded as a first size, the pixel arrangement specification of the image supported by the electronic device for display is recorded as a second size, and the camera sensor of the electronic device supports full-size image output at a first zoom factor.

[0096] When the electronic device captures an image at a first zoom ratio, the electronic device controls the camera sensor to output the image at full size. When the electronic device captures an image at a target zoom ratio, the electronic device reuses the function of the camera sensor to output the image at full size at the first zoom ratio, controls the camera sensor to output the first image at full size and the second image at a combined size, and the image processor of the electronic device obtains a new image based on the first image and the second image, which serves as the image corresponding to the current zoom ratio and can include both a wider background and a clearer partial pixel area. In this way, the electronic device does not need to be equipped with a telephoto lens, and can obtain an image that includes both a wider background and a clearer image using only the main camera lens.

[0097] The image processing method provided in this embodiment is applied to electronic devices, which may include personal computers (PCs), tablet computers, laptops, portable computers (such as mobile phones), wearable electronic devices (such as smart watches), augmented reality (AR) and virtual reality (VR) devices, car computers, and other electronic devices with cameras. The following embodiments do not impose any special restrictions on the specific form of the electronic devices.

[0098] like Figure 4 FIG. 1 is a flow chart of an image processing method provided in this embodiment. The provided image processing method mainly includes the following processes:

[0099] S41: The electronic device displays a shooting preview interface; the shooting preview interface includes a preview image corresponding to the target zoom ratio and the framing range of the main camera lens.

[0100] The image processing method provided in this embodiment is applied to an electronic device, and the electronic device is installed with a related application program capable of executing the provided image processing method.

[0101] In one example, a third-party image processing application is installed in the electronic device to perform the provided image processing operation. The electronic device displays an icon for the image processing application on the desktop. When the electronic device receives a click operation on the image processing application icon, it begins to perform the image processing operation corresponding to the provided image processing method.

[0102] In another example, an electronic device is installed with a camera application that operates in multiple shooting modes, such as photo, portrait, panorama, professional, and video shooting modes. When the camera application operates in one or more shooting modes, for example, when the camera operates in photo mode, the electronic device begins executing the provided image processing method. The specific implementation of the image processing method provided in this embodiment will be explained below using the photo mode within the camera application as an example.

[0103] For example Figure 5 As shown in (1), the electronic device displays a shooting preview interface after opening the camera application, and the current shooting mode is the photo mode (such as Figure 5 In the photo mode, the camera lens called by the camera application is the main camera lens, and the main camera lens and the camera sensor cooperate to capture the image within the framing range of the main camera lens, and the preview image within the framing range of the main camera lens is updated in real time in the form of a preview stream in the shooting preview interface (such as Figure 5 B) shown in (1) in the figure.

[0104] Figure 5 As shown in (1), the shooting preview interface displayed by the electronic device also includes a zoom control (such as Figure 5 (C) shown in (1) of FIG). If the electronic device does not receive a click operation on the zoom control, the default zoom ratio is 1x, that is, the target zoom ratio is 1x. If the electronic device receives a click operation on the zoom control, the electronic device confirms the target zoom ratio selected by the user based on the click operation on the zoom control.

[0105] Based on the aforementioned background information, the images captured by the main camera lens and camera sensor of an electronic device are related to the zoom factor. When the electronic device updates and displays the preview image in the form of a preview stream on the shooting preview interface, if the target zoom factor changes, the preview image displayed on the shooting preview interface will also change accordingly.

[0106] S42: In response to a photo-taking operation, the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to the image processor respectively; wherein the first image is an image output by the camera sensor in a full-size output mode, and the second image is an image output by the camera sensor in a merged-size output mode according to a target zoom ratio.

[0107] When the electronic device is updating and displaying a preview image on a shooting preview interface, if it receives a photo-taking operation from a user, the electronic device performs image acquisition and processing operations in response to the photo-taking operation.

[0108] In specific implementation, the photo operation can be a click operation on the photo control in the shooting preview interface collected by the electronic device, or the photo operation can also be a voice control operation containing keywords such as "photo" collected by the electronic device, etc., without limitation.

[0109] like Figure 5 As shown in (2) in FIG, the electronic device responds to the user applying the photo control (such as Figure 5 The click operation (D) in (2) opens the shutter, and light within the main camera lens's framing range passes through the open shutter and converges onto the camera sensor's light-receiving surface. Each pixel in the camera sensor senses the light signal and converts it into a corresponding electrical signal.

[0110] In one example, the first size of the pixel arrangement specification within the camera sensor of the electronic device is 9000*12000, the second size of the pixel arrangement specification of the image supported by the electronic device for display is 3000*4000, and the camera sensor of the electronic device supports full-size image output at a first zoom ratio, i.e., 3x.

[0111] In this embodiment, the electronic device captures images based on a target zoom ratio. When the camera sensor reads the electrical signals converted by each pixel and obtains pixel data for image output, it performs two image output operations. The sizes of the images output by the two image output operations are different.

[0112] In a specific embodiment, the first image output operation performed by the camera sensor is to output a first image of a first size in a full-size output manner.

[0113] like Figure 5 As shown in (3), the camera sensor reuses the full-size image output function at the target zoom ratio at 3x magnification, and outputs the first image of the first size in the full-size output mode at the target zoom ratio.

[0114] In another specific embodiment, the second image output operation performed by the camera sensor is to output a second image of a second size in a merged size output manner.

[0115] like Figure 5 As shown in (3), the camera sensor retains the operation of merging the size and outputting the image at the target zoom ratio, and reads the pixel data at the target zoom ratio to obtain a second image of the second size.

[0116] That is to say, when the electronic device captures images at a target zoom ratio, unlike the existing solution of outputting one frame of image to the image processor at one zoom ratio, the camera sensor in this embodiment outputs two frames of images of different sizes to the image processor in two different output modes.

[0117] In a specific example, the target zoom ratio is 1x, and the specific implementation scheme of the camera sensor combining the size and reading the pixel data to obtain the second image is shown in the figure. Figure 3 As shown in (1), it is the full-size 9000*12000 image converted from the camera sensor. Figure 3 As shown in (2), the 9000*12000 pixel array is divided into three parts to obtain pixel areas Z1, Z2...Z11 and Z12. The pixel areas are merged and read to obtain the corresponding pixel areas A1, A2...A11 and A12, as shown in Figure 3 As shown in (3) in the figure, the second image obtained after merging and reading pixel data is as follows Figure 3 As shown in (4) in the figure. The camera sensor is at 1x magnification. Figure 3 The specific implementation method of the camera sensor merging size to output the second image with a 1x magnification can be found in the aforementioned Figure 3 The images and text descriptions shown will not be repeated here.

[0118] In another specific example, if the target zoom ratio is 2x, the specific implementation scheme of the camera sensor combining the size and reading the pixel data to obtain the second image is as follows: Figure 6 As shown. Figure 6 As shown in (1), it is the full-size image converted by the camera sensor, and the camera sensor merges and reads the pixel data. The camera sensor first merges and reads the pixel data in a three-in-one manner, that is, the full-size image P1 of 9000*12000 is converted into a full-size image P1 of 9000*12000. Figure 6 As shown in (2) in , the merged size is 3000*4000 image P2. Figure 6 As shown in (3) in , the image P2 is magnified by 2 times to obtain the image P3, as shown in Figure 6 As shown in (4), image P3 is then cropped from the center area to obtain image P4. Image P4, obtained after sequentially merging, reading, amplifying, and cropping, is the second image output by the camera sensor at a 2x magnification.

[0119] In another specific example, the target zoom ratio is 3x, and the camera sensor combines the size and reads the pixel data to obtain the specific implementation scheme of the second image as follows: Figure 2 As shown. Figure 2 As shown in (2), the camera sensor outputs a full-size image to the image processor.

[0120] The above example uses a camera sensor with a pixel size of 108M, an image size of 12M, and a first zoom ratio of 3x to support full-size output as an example to explain the specific implementation scheme for an electronic device to control the camera sensor to output the first and second images. In this case, the camera sensor outputs a 108M first image and a 12M second image corresponding to the target zoom ratio. At a 1x zoom ratio, the camera sensor outputs the second image using a three-in-one combined size.

[0121] In other cases, the camera sensor pixel size and image size may also be different.

[0122] For example, in one scenario, the camera sensor has a pixel size of 50M and an image size of 12.5M. The first zoom ratio that the camera sensor supports for full-size output is 2x. At a 1x zoom ratio, the camera outputs the 50M first image at full size, and the camera sensor outputs the 12.5M second image at a two-in-one combined size.

[0123] For example, in another scenario, the camera sensor has a pixel size of 200M and an image size of 12.5M. The first zoom ratio supported by the camera sensor for full-size output is 4x. At a 1x zoom ratio, the camera outputs a 200M first image at full size, and the camera sensor outputs a 12.5M second image using a four-in-one binning scheme.

[0124] Typically, the pixel and image specifications of a camera sensor within an electronic device are fixed, and the first zoom ratio supporting full-size image output is also fixed. The pixel and image specifications, and first zoom ratios, of different electronic devices may be the same or different. In specific implementations, each electronic device may adaptively adjust the execution of the response steps based on the pixel and image specifications of its camera sensor, without limitation.

[0125] S43: The electronic device fuses the first image and the second image to obtain a third image; wherein the size of the third image is the second size, and the clarity of at least part of the pixel area of ​​the third image is higher than the clarity of the second image.

[0126] The camera sensor of the electronic device outputs two images of different sizes to an image processor. The image processor generates a third image based on the two images of different sizes as the final third image obtained in response to a user's photo taking operation. The zoom ratio of the third image generated by the electronic device is the target zoom ratio.

[0127] While updating the capture preview interface before receiving a capture operation, the electronic device may receive multiple zoom operations initiated by the user, and different zoom operations may indicate different zoom factors. The electronic device uses the last zoom factor determined before receiving the capture operation as the target zoom factor and generates a third image corresponding to the target zoom factor.

[0128] In one example, the electronic device can directly display the generated third image on the interface of a camera application or a third-party application. The electronic device can also display a save control on the interface, and the electronic device can save the third image to an album in response to a user clicking the save control. The electronic device can also display a cancel control on the interface, and the electronic device can delete the generated third image in response to a user clicking the cancel control.

[0129] In another example, the electronic device may directly save the generated third image to an album. Furthermore, the electronic device may continue to display the shooting preview interface and update the thumbnail of the image previously displayed in the album display area within the shooting preview interface to a thumbnail of the third image. If the electronic device receives a click operation on the album display area within the shooting preview interface, the electronic device may switch the interface to display the third image.

[0130] In a specific embodiment, the step of the electronic device generating the third image may further include:

[0131] S431: The electronic device crops the first image to obtain a fourth image; the size of the fourth image is the third size, and the third size is smaller than or equal to the second size.

[0132] S432: The electronic device fuses the second image and the fourth image to obtain a third image.

[0133] The first size of the first image output by the camera sensor is larger than the second size supported by the electronic device. The image processor can first crop or downsample the first image to obtain a fourth image of a relatively smaller size. The image processor then generates a third image based on the fourth image and the second image. The third size of the cropped fourth image is less than or equal to the second size.

[0134] There are many specific solutions for the image processor to crop the first image to obtain the fourth image.

[0135] In one example, the image processor crops a pixel region at fixed coordinates in the first image to obtain the fourth image. For example, the image processor crops a central pixel region at fixed coordinates in the first image to obtain the fourth image.

[0136] In this case, the image processor directly crops the central pixel area of ​​the first image to obtain the fourth image. The image processor directly crops the central pixel area of ​​the full-size first image to obtain the fourth image. Compared with the central pixel area of ​​the second image obtained by combining the sizes, the fourth image has the same characteristic objects but higher definition.

[0137] For example Figure 2 As shown in (3), the image processor crops the central pixel area of ​​the first image to obtain Figure 2 The image shown in (4) in .

[0138] In another example, the image processor crops pixel regions with unfixed coordinates in the first image to obtain the fourth image. For example, the image processor may crop based on the pixel region where the target feature object is located in the first image. The coordinates of the pixel region of the target feature object in the first image may not be fixed, and the coordinates of the pixel regions to be cropped may also not be fixed in different situations.

[0139] When cropping the first image, the image processor may crop based on the pixel region where the target feature object is located. The target feature object may be a human face, an animal face, a flower face, or other types of feature objects, which are typically the features that a user may focus on. When cropping the first image, the image processor crops the pixel region where the target feature object is located to obtain the fourth image, thereby ensuring that the pixel region where the feature object of interest to the user is located has higher clarity.

[0140] The electronic device executes the step S431, which may specifically include:

[0141] S4311: The electronic device determines the real-time coordinate range of the target feature object in the world coordinate system.

[0142] S4312: The electronic device maps the real-time coordinate range to the first image according to the mapping relationship between the world coordinate system and the image coordinate system, and determines the first pixel area where the target feature object is located.

[0143] The electronic device may store a feature recognition algorithm, use the feature recognition algorithm to identify a target feature object, and determine the real-time coordinate range of the target feature object in the world coordinate system. The following uses a face as an example to explain the target feature object recognition scheme.

[0144] Feature recognition algorithms stored in electronic devices may include feature extraction algorithms and face recognition algorithms. Feature extraction algorithms identify a person's face by extracting key features (such as the coordinates, outlines, and colors of the eyes, nose, and mouth), modeling facial features using statistical models (such as Gaussian mixture models, principal component analysis, and linear discriminant analysis), and then using image transformation algorithms to extract features through image transformations (such as integral images and wavelet transforms).

[0145] Face recognition algorithms can include Eigen Face, Fisher Face, and Multi-Task Convolutional Neural Network (MTCNN), which are used for facial feature extraction and face detection, respectively. The facial feature recognition algorithm provided in this embodiment can be other commonly used face recognition algorithms, without limitation.

[0146] Similarly, when the target feature object is a feature object of the type of animal face, flower, etc., the electronic device can also use the feature extraction algorithm and recognition algorithm commonly used in this field to identify the target feature object within the viewing range and determine the real-time coordinate range of the target feature object.

[0147] In one case, the electronic device uses a feature recognition algorithm to detect in real time whether each preview image in the preview stream includes a target feature object in a shooting preview scene. If the target feature object is included, the coordinate range of the target feature object is determined and the coordinate range of the target feature object is fed back to the image processor.

[0148] In this way, after obtaining the first image output by the camera sensor, the image processor can map the real-time coordinate range to the first image according to the mapping relationship between the world coordinate system and the image coordinate system, and determine the first pixel area where the target feature object is located.

[0149] When a photo-taking application is running, the electronic device can continuously suspend the face recognition algorithm so as to identify the face object and detect the area where the face object is located in real time during the photo-taking process.

[0150] In other cases, the electronic device is loaded with a feature recognition algorithm, and uses the feature recognition algorithm to recognize the target feature object in the first image and determine the first pixel area of ​​the target feature object in the first image.

[0151] S4313: The electronic device cuts out a block including the first pixel area from the first image as a fourth image.

[0152] The electronic device determines the first pixel area of ​​the target feature object in the first image based on the above steps, and can crop the first image to obtain a fourth image including the first pixel area.

[0153] In one example, the electronic device may directly perform cropping using the first cropping frame corresponding to the first pixel region, and the pixel features contained in the cropped fourth image are consistent with the pixel features contained in the first pixel region, and the pixel sizes are consistent.

[0154] For example Figure 7 As shown in FIG, it is a schematic diagram of cropping the first image to obtain the fourth image. Figure 7 (1) in FIG. 1 is a schematic diagram of the first image. The first cropping frame S1 is determined based on the first pixel area Z1 in the first image. The fourth image is obtained by cropping with the first cropping frame S1. Figure 7 As shown in (2) in FIG. 1 , it can be seen that the pixel features included in the fourth image are consistent with the pixel features included in the first pixel area Z1 , and the pixel sizes are consistent.

[0155] In another example, the step of the electronic device cropping the region including the first pixel from the first image may also include the following steps:

[0156] Determining a target cropping frame based on a central pixel point of the first pixel region, wherein the central point of the target cropping frame coincides with the central pixel point of the first pixel region, the size of the target cropping frame is a second size, and a pixel region covered by the target cropping frame at least partially coincides with the first pixel region;

[0157] The fourth image is cropped from the first image according to the target cropping frame.

[0158] In this example, the electronic device may first determine a target cropping frame of the second size based on the first pixel region, and then crop the first image based on the target cropping frame to obtain a fourth image, where the third size of the fourth image is consistent with the second size. The pixel features included in the fourth image may not be completely consistent with the pixel features included in the first pixel region, that is, they may at least partially overlap.

[0159] For example Figure 8 As shown in FIG, it is another schematic diagram of cropping the first image to obtain the fourth image. Figure 8 (1) is a schematic diagram of a first image, and a target cropping frame S2 is determined according to a first pixel area Z1 in the first image. Figure 8 In (1), the pixel area covered by the target cropping frame S2 completely includes the first pixel area.

[0160] The image processor crops the first image according to the target cropping frame S2 to obtain a fourth image, such as Figure 8As shown in (2) in . It can be seen that the pixel features contained in the fourth image include the pixel features contained in the first pixel area Z1, and the pixel size of the fourth image is consistent with the pixel size of the second image. In this way, when the image processor performs the fusion processing on the second image and the fourth image, since the pixel sizes of the two frames are consistent, the image processor does not need to perform the zero padding and size alignment operation on the fourth image.

[0161] Based on the aforementioned steps, the electronic device obtains a second image and a fourth image. The second image is an image corresponding to the target zoom factor, includes the target feature object and a wider background, and the pixel area where the target feature object is located has a lower definition. The fourth image includes the target feature object, and the pixel area where the target feature object is located has a higher definition.

[0162] The electronic device executes S432 to fuse the second image and the fourth image to obtain the third image. This is actually to fuse the pixel features of the target feature object with higher definition, included in the fourth image, into the second image to enhance the definition of the target feature object in the second image. Thus, in the third image obtained by the fusion process, the pixel region containing the target feature object has higher definition than the pixel region of the target feature object in the second image.

[0163] The electronic device executes S432 to fuse the second image and the fourth image to obtain the third image. There are many implementation schemes.

[0164] In one example, the electronic device may utilize an image fusion algorithm to fuse pixel features in the fourth image into corresponding pixel regions in the second image to obtain a third image.

[0165] In another example, the electronic device may also load a pre-trained image processing model. When the electronic device generates the third image, it may also use the image processing model to process the second image and the fourth image to obtain the third image.

[0166] The model is essentially a trained neural network. A neural network is a network formed by connecting multiple individual neural units. The output of one neural unit can be the input of one or more other neural units. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from that local receptive field, which can be an area consisting of several neural units.

[0167] Neural networks used to train image processing models typically include deep neural networks (DNNs) or convolutional neural networks (CNNs). Deep neural networks, also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. Based on the location of different layers, DNNs can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. Layers can be fully connected. In other words, any neuron in layer i can be connected to any neuron in layer i+1.

[0168] A convolutional neural network is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of a convolutional layer and a subsampling layer, which can be considered a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. Within a convolutional layer, a neuron can only connect to some of the neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neural units.

[0169] Model training for electronic devices typically requires multiple sets of sample images. Each set of sample images consists of two types of images: input sample images containing features to be classified, and target sample images containing target features. It should be noted that the features to be classified here refer to features that require processing operations such as recognition, classification, and labeling, such as features with clear or blurred faces. Target features are one or more categories of features that need to be identified or processed, such as features with fused images or clarity. The image content of the input sample images and target sample images is correlated.

[0170] The input sample image serves as the input value of the neural network, and the target sample image serves as the target value of the neural network. The input sample image contains various basic elements that may be included in the image to be processed by the image processing model. The target sample image can be an image obtained by processing the input sample image according to user requirements. The target sample image may be an image obtained by processing the input sample image according to user requirements, and the target sample image may remove elements that the user does not need based on the input sample image. In other words, the element content of the input sample image and the target sample image is basically the same, but the feature attributes may be different.

[0171] In this embodiment, the electronic device can train a neural network to obtain an image processing model, and then use the image processing model to perform image processing operations. The electronic device can also directly obtain an image processing model trained by other devices to perform the image processing method provided in this embodiment.

[0172] The process of an electronic device training a neural network to obtain an image processing model mainly includes: the electronic device obtains a sample image and a neural network without setting relevant parameters of the weight matrix, inputs the sample image into the neural network for iterative training, and gradually optimizes the weight matrix in the neural network according to the loss function until the loss function converges, so that the weight matrix in the neural network reaches the optimal value. At this time, the neural network can be used as an image processing model.

[0173] The sample images acquired by the electronic device include an input sample image and a target sample image. The input sample image may include a first sample image and a second sample image, and the first sample image and the second sample image both have a second pixel size. The first sample image includes a wider background and a sample feature object, and the pixel area where the sample feature object is located in the first sample image has a lower definition. The second sample image includes the sample feature object, and the pixel area where the sample feature object is located in the second sample image has a higher definition.

[0174] The target sample image also has a pixel size of the second size. The target feature object includes the sample feature object and a broader background. The pixel region where the sample feature object resides in the target sample image has a higher definition. Compared to the first sample image, the pixel region where the sample feature object resides in the target sample image has a higher definition.

[0175] The electronic device uses the first and second sample images as input values ​​and the target sample training as the target value to iteratively train a neural network without set weight parameters. Specifically, the electronic device uses a loss function to calculate the difference between the predicted value and the true value, and then uses the difference backpropagation to modify the weight matrix. The electronic device then inputs the training sample data into the neural network with the modified weight matrix, and performs multiple iterative training until the difference is minimized, indicating that the neural network converges.

[0176] When a neural network is iteratively trained using input values ​​and target values, an error backpropagation algorithm is used to correct the parameters of the initial neural network weight matrix during training, reducing the neural network's reconstruction error loss. Specifically, the difference between the predicted value and the target value obtained in each iterative training is calculated, and the difference is then forwarded back to the neural network to modify the neural network's weight matrix. The neural network with the modified weight matrix is ​​then iteratively trained again, and the difference obtained from the further iterative training is used to continue updating the neural network's weight matrix. Repeat multiple iterative trainings and reversely modify the weight matrix using the difference from each iterative training until the difference between the predicted value and the target value obtained is within the allowable loss range of the loss function, that is, the loss function converges. This means that the neural network has achieved convergence through multiple iterative trainings. The backpropagation algorithm is a backpropagation movement dominated by error loss, used to obtain the optimal parameters of the neural network model. Of course, iterative training can also be iteratively optimized using other similar algorithms, without limitation.

[0177] The electronic device trains an image processing model according to the above scheme and stores the trained image processing model. When the image processor performs fusion processing on the second image and the fourth image, the second image and the fourth image can be input into the image processing model for image processing, and the image output by the image processing model is used as the third image.

[0178] like Figure 9 FIG2 is a process diagram of an image processing method. When the electronic device performs a photo-taking operation at a target zoom ratio, the camera sensor is controlled to output a first image of a first size and a second image of a second size. The first image of the first size is cropped according to the first pixel area where the target feature object is located to obtain a fourth image. The electronic device inputs the second image and the fourth image into an image processing model to obtain a third image. The third image includes both the wider background corresponding to the viewing range of the main camera lens at the target zoom ratio and the relatively clear target feature object, resulting in a higher image quality.

[0179] In a specific embodiment, another image processing method is provided. The difference between the provided image processing method and the above image processing method is that the electronic device can select different execution schemes according to the relationship between the target zoom ratio and the first zoom ratio.

[0180] Specifically, such as Figure 10 As shown, the provided image processing method mainly includes the following steps:

[0181] S1001, the electronic device determines whether the target zoom ratio is a first zoom ratio;

[0182] If the target zoom ratio is the first zoom ratio, executing S1002, the electronic device controls the camera sensor to output a first image of a first size to the image processor in response to the photographing operation;

[0183] S1003: The electronic device generates a fifth image; wherein the fifth image is an image obtained by the image processor by cropping the first image.

[0184] If the target zoom ratio is not the first zoom ratio, step S1004 is executed. In response to the photographing operation, the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to the image processor.

[0185] S1005: The electronic device generates a third image according to the first image and the second image.

[0186] The camera sensor of the electronic device has the ability to output full-size images at a first zoom ratio. In order to save computing resources, the electronic device can distinguish between the cases where the target zoom ratio is the first zoom ratio and the cases where the target zoom ratio is not the first zoom ratio, and perform different image processing operations respectively.

[0187] In one case, the target zoom ratio is not the first zoom ratio. The electronic device can control the camera sensor to output a first image of a first size and a second image of a second size to the image processor respectively. The specific implementation process can refer to the aforementioned Figure 4 The specific implementation process of the image processing method shown will not be described in detail here.

[0188] In another case, the target zoom ratio is the first zoom ratio, for example, for a 108M camera sensor, the target zoom ratio is 3x.

[0189] In this case, the camera sensor directly outputs the full-size first image to the image processor at the 3x magnification based on its full-size image output capability. The image processor crops the first image, for example Figure 2 The fifth image is obtained by cropping the central pixel area shown in (3). Since the full-size image has a high definition, the definition of the cropped fifth image is also relatively high. That is, a high-quality image with high definition can be obtained without collecting and fusing multiple frames of images.

[0190] In this embodiment, the electronic device distinguishes between the case where the target zoom magnification is the first zoom magnification and the case where the target zoom magnification is not the first zoom magnification, which can save computing resources and obtain high-quality images.

[0191] In other embodiments, the electronic device may also distinguish the brightness of the current shooting environment and select different processing schemes. When the shooting environment is bright, the camera sensor outputs the first image in full size, with a high signal-to-noise ratio and good image quality. When the shooting environment is dim, the camera sensor outputs the first image in full size, with a low signal-to-noise ratio and poor image quality. The quality of the third image generated based on the poor first image is also poor.

[0192] Based on this, this embodiment can also limit the prerequisite for performing the image processing operation to a high-brightness shooting environment. The details are as follows:

[0193] The electronic device detects whether the current shooting mode is a low-light shooting mode or a night scene shooting mode;

[0194] If the current shooting environment is a low-light shooting mode or a night scene shooting mode, control the camera sensor to output a second image of a second size according to the target zoom ratio to the image processor;

[0195] displaying a second image;

[0196] If the target zoom ratio is not the low-light shooting mode or the night scene shooting mode, the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to the image processor in response to the photographing operation.

[0197] In one case, the electronic device may be limited to not executing the image processing operation provided in this embodiment in the night scene mode.

[0198] In another case, the electronic device can also call the ambient light sensor to collect the ambient light brightness of the shooting environment when the camera application is turned on. If the ambient light brightness meets the brightness requirements of a high-brightness environment, the image processing method provided in this embodiment can be executed to obtain a high-quality image. If the ambient light brightness does not meet the brightness requirements of a high-brightness environment, the image processing method provided in this embodiment may not be executed.

[0199] The image processing method provided in the above embodiment can obtain images that simultaneously include a wider background and a clearer face image based on the camera's main camera lens. When the target zoom ratio is not the first zoom ratio, the camera sensor reuses the on-chip zoom (Insensor Zoom) capability at the first zoom ratio. This expands the cropped position after full-size image output from the central pixel area to the pixel area guided by the user's focus, providing 3X HD image quality for the user's area of ​​interest. Subsequent image fusion enables even higher clarity for the area the user is most focused on, achieving zoom fusion based on a single camera and resolving the dependency on the telephoto lens.

[0200] The above embodiment explains the specific implementation process of the image processing method from the perspective of the electronic device. The following will explain the specific implementation process of the image processing method from the perspective of the internal software architecture of the electronic device.

[0201] like Figure 11 The following is a detailed explanation of the image processing process of the electronic device in combination with the internal architecture of the electronic device.

[0202] Specifically, the internal architecture of an electronic device can be divided into four layers: the application layer (APP), the framework layer (FWK), the hardware abstraction layer (HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included without limitation.

[0203] The application layer can include a series of application packages, such as the camera application involved in this embodiment. In addition, the application layer also includes applications such as gallery and image processing applications with camera functions. Application packages can also include applications such as call, calendar, map, navigation, music, video, short message, etc.

[0204] The framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The framework layer includes some predefined functions.

[0205] The framework layer runs a camera service, which can be called by camera applications to implement photography-related functions. In addition, the framework layer may also include a window manager, content providers, a view system, a resource manager, and a notification manager. The window manager manages window applications. It can obtain the display size, determine whether a status bar is present, lock the screen, and take screenshots. The content provider stores and retrieves data and makes it accessible to applications. This data can include video, images, and audio. The view system includes visual controls, such as those for displaying text or images. The view system is used to build applications. The display interface can consist of one or more views. The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager enables applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, requiring no user interaction. For example, notifications are used to announce download completions and message reminders. Notifications can also appear as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog windows that appear on the screen. Notification information may also be, for example, a text message displayed in the status bar, a beep sound, vibration of the terminal, flashing indicator light, etc. It should be noted that the camera application may also call content providers, resource managers, notification managers, window managers, view systems, etc. according to actual business needs, and this embodiment does not impose any restrictions on this.

[0206] The kernel layer is a layer between hardware and software. The kernel layer includes at least a camera driver. The camera driver can be used to drive a hardware module with a shooting function, such as a camera sensor. In other words, the camera driver is responsible for data exchange with the camera sensor. The kernel layer may also include a display driver, an audio driver, a sensor driver, etc., which is not limited in this embodiment. In this solution, the electronic device includes a camera, the camera includes a main camera lens, the camera driver is used to drive the main camera lens to open the shutter and adjust the lens focal length of the lens combination, and the camera sensor exchanges data with the camera driver to realize the image acquisition function of the camera.

[0207] The hardware abstraction layer can encapsulate the drivers in the kernel layer and provide a calling interface to the framework layer, shielding the implementation details of the underlying hardware. Figure 11As shown, the hardware abstraction layer can include, among other things, a camera call processing module (Camera HAL). The camera call processing module is the core software framework for the camera, and includes, among other things, an interface module, a sensor call module (Sensor Node), and an image processing module. These interface module, sensor call module, and image processing module are components within the camera call processing module's image data and control instruction transmission pipeline. The image processing module can call the image processor to implement the aforementioned image processing functions.

[0208] Specifically, the sensor module can be a control node for the camera sensor, which can control the camera sensor through the camera driver. The interface module can be a software interface for the application framework layer, used to exchange data with the application framework layer. Of course, the interface module can also exchange data with the image processing module in the camera call processing module.

[0209] The image processing module can process the first image and the second image output by the camera sensor, crop the first image to obtain a fourth image, and fuse the second image and the fourth image to obtain a third image.

[0210] like Figure 11 As shown, at the application layer, the camera application can receive the user's zoom operation and pass the target zoom ratio selected by the user to the camera service of the framework layer.

[0211] At the framework layer, the camera service receives the target zoom ratio and generates a photo request, which indicates the target zoom ratio and the way in which the camera sensor outputs the image: outputting the first image in full size and / or outputting the second image in a merged size.

[0212] The camera service stores the first zoom ratio. If it is determined that the target zoom ratio is not the first zoom ratio, the generated photo request instructs the camera sensor to output the image in the following manner: output the first image in full size, and output the second image in merged size.

[0213] If the camera service determines that the target zoom ratio is the first zoom ratio, the generated photo request instructs the camera sensor to output the image in a manner of outputting the first image in full size.

[0214] The camera service sends the photo request to the hardware abstraction layer through the application program interface.

[0215] At the hardware abstraction layer, the interface module receives the photo request and passes the photo request to the sensor calling module and the image processing module respectively.

[0216] The sensor calling module can call the camera sensor through the camera driver, so that the camera sensor outputs a first image of a first size and a second image of a second size respectively. The camera sensor outputs the first image of the first size in a full-size output manner.

[0217] The camera sensor first downsamples the image of the second size in a merged size output manner, then magnifies the image of the second size according to the zoom ratio, and then crops the image according to the central pixel area to obtain a second image of the second size.

[0218] The image processing module receives the first image and crops the first image to obtain a fourth image. Specifically, the image processing module first determines a first pixel region in the first image based on the facial coordinates, crops the first image based on the first pixel region, and obtains a new image as the fourth image.

[0219] The image processing module receives the second image, and fuses the second image and the fourth image to obtain a third image.

[0220] The image processing module uploads the third image (which can be uploaded through the interface module) to the framework layer, and the camera service and view system of the framework layer render the third image, and then upload it to the application layer for display or storage.

[0221] It is understandable that Figure 11 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown, and each layer may include more or fewer components, and this application does not limit this.

[0222] In addition, it is understood that in order to implement the image processing method in this embodiment, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0223] In addition, an embodiment of the present application further provides an electronic device, including a camera, a memory, and a processor;

[0224] Memory stores computer-executable instructions;

[0225] The processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the image processing method provided by the above embodiment. In addition to these main components, the electronic device also includes components for realizing basic functions. Figure 12 Provide specific instructions.

[0226] like Figure 12 The figure shows a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of the present application. The electronic device 1200 may include a processor 1210, an external memory interface 1220, an internal memory 1221, a Universal Serial Bus (USB) interface 1230, a charging management module 1240, a power management module 1241, a battery 1242, an antenna 1, an antenna 2, a mobile communication module 1250, a wireless communication module 1260, an audio module 1270, a speaker, a receiver, a microphone, an earphone interface, a sensor module 1280, a button 1290, a motor 1291, an indicator 1292, a camera 1293, a display 1294, and a SIM card module 1295. The sensor module 1280 may include a pressure sensor 1280A, a gyroscope sensor 1280B, an ambient light sensor 1280C, a touch sensor 1280D, and the like.

[0227] The illustrated structure of the embodiment of the present invention does not limit the electronic device 1200. It may include more or fewer components than shown, or some components may be combined or separated, or arranged differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0228] The processor 1210 may include one or more processing units. For example, the processor 1210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0229] The controller is the decision-maker that directs the various components of electronic device 1200 to coordinate operations according to instructions. It serves as the nerve center and command center of electronic device 1200. Based on instruction opcodes and timing signals, the controller generates operational control signals to control instruction fetching and execution.

[0230] Processor 1210 may also include memory for storing instructions and data. In some embodiments, the memory in processor 1210 is a high-speed cache memory that can store instructions or data that have just been used or are being recycled by processor 1210. If processor 1210 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 1210 latency, and thus improves system efficiency.

[0231] In some embodiments, the processor 1210 may include an interface. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface.

[0232] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDL) and a serial clock line (SCL). In some embodiments, the processor 1210 may include multiple I2C bus lines. The processor 1210 may be coupled to the touch sensor 1280D, charger, flash, camera 1293, etc. via different I2C bus interfaces. For example, the processor 1210 may be coupled to the touch sensor 1280D via the I2C interface, enabling communication between the processor 1210 and the touch sensor 1280D via the I2C bus interface, thereby implementing the touch function of the electronic device 1200.

[0233] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1210 to the camera 1293, the display 1294, the wireless communication module 1260, the audio module 1270, the sensor module 1280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0234] USB interface 1230 can be a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. USB interface 1230 can be used to connect a charger to charge electronic device 1200, or to transfer data between electronic device 1200 and peripheral devices. It can also be used to connect headphones to play audio. It can also be used to connect other electronic devices, such as AR devices.

[0235] The interface connection relationship between the modules in the embodiment of the present invention is only for illustrative purposes and does not limit the structure of the electronic device 1200. The electronic device 1200 can adopt different interface connection methods in the embodiment of the present invention, or a combination of multiple interface connection methods.

[0236] The charging management module 1240 can be a rechargeable battery or a disposable battery. For rechargeable batteries, it can receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1240 can receive charging input from a wired charger via the USB interface 1230. In some wireless charging embodiments, the charging management module 1240 can receive wireless charging input via the wireless charging coil of the electronic device 1200. While the charging management module 1240 is charging the battery 1242, it can also provide power to the electronic device 1200 through the power management module 1241.

[0237] The power management module 1241 is used to connect the battery 1242, the charging management module 1240, and the processor 1210. The power management module 1241 receives input from the battery 1242 and / or the charging management module 1240 and provides power to the processor 1210, the memory 1221, the external memory interface 1220, the display 1294, the camera 1293, and the wireless communication module 1260. The power management module 1241 can also be used to monitor parameters such as the charging management module capacity, the number of charging management module cycles, and the health status (leakage, impedance) of the charging management module. In some embodiments, the power management module 1241 can also be provided in the processor 1210. In some embodiments, the power management module 1241 and the battery 1242 can also be provided in the same device.

[0238] The wireless communication function of the electronic device 1200 can be implemented through antenna 1, antenna 2, mobile communication module 1250, wireless communication module 1260, modem and baseband processor.

[0239] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in conjunction with a tuning switch.

[0240] The mobile communication module 1250 can provide a communication processing module for wireless communication solutions such as 2G / 3G / 4G / 5G applied to the electronic device 1200. The mobile communication module 1250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 1250 receives electromagnetic waves from the antenna 1, filters and amplifies the received electromagnetic waves, and transmits them to the modem for demodulation. The mobile communication module 1250 can also amplify the signal modulated by the modem and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 1250 can be set in the processor 1210. In some embodiments, at least some of the functional modules of the mobile communication module 1250 can be set in the same device as at least some of the modules of the processor 1210.

[0241] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.) or displays an image or video through the display screen 1294. In some embodiments, the modem can be an independent device. In some embodiments, the modem can be independent of the processor 1210 and be set in the same device as the mobile communication module 1250 or other functional modules.

[0242] The wireless communication module 1260 can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc., which are applied to the electronic device 1200. The wireless communication module 1260 can be one or more devices that integrate at least one communication processing module. The wireless communication module 1260 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 1210. The wireless communication module 1260 can also receive the signal to be sent from the processor 1210, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0243] In some embodiments, antenna 1 of electronic device 1200 is coupled to mobile communication module 1250, and antenna 2 is coupled to wireless communication module 1260, so that electronic device 1200 can communicate with a network and other devices via wireless communication technologies. Wireless communication technologies may include Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. GNSS may include Global Positioning System (Satellite Based Augmentation Systems, SBAS), Global Navigation Satellite System (GLONASS), BeiDou navigation Satellite system (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation System (SBAS).

[0244] Electronic device 1200 implements display functionality through a GPU, display screen 1294, and an application processor. The GPU is a microprocessor for image processing that connects display screen 1294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 1210 may include one or more GPUs that execute program instructions to generate or modify display information.

[0245] Display screen 1294 is used to display images, videos, etc. Display screen 1294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 1200 may include one or N display screens 1294, where N is a positive integer greater than 1.

[0246] The electronic device 1200 can implement a shooting function through an ISP, a camera 1293, a video codec, a GPU, a display screen, and an application processor.

[0247] The ISP processes data fed back by the camera 1293. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within the camera 1293.

[0248] The camera 1293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 1200 may include 1 or N cameras 1293, where N is a positive integer greater than 1.

[0249] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 1200 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0250] Video codecs are used to compress or decompress digital video. Electronic device 1200 may support one or more video codecs. This allows electronic device 1200 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0251] The NPU is a neural network (NN) computing processor that rapidly processes input information and continuously self-learns by drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain. The NPU can enable intelligent cognitive applications in electronic device 1200, such as image recognition, face recognition, speech recognition, and text comprehension.

[0252] The external memory interface 1220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1200. The external memory card communicates with the processor 1210 via the external memory interface 1220 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0253] The internal memory 1221 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 1210 executes various functional applications and data processing of the electronic device 1200 by running the instructions stored in the internal memory 1221. The memory 1221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 1200 (such as audio data, a phone book, etc.), etc. In addition, the memory 1221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash storage (Universal Flash Storage, UFS), etc.

[0254] The electronic device 1200 can implement audio functions such as music playback and recording through the audio module 1270, speakers, receivers, microphones, headphone jacks, and application processors.

[0255] The audio module 1270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 1270 can also be used to encode and decode audio signals. In some embodiments, the audio module 1270 can be provided in the processor 1210, or some functional modules of the audio module 1270 can be provided in the processor 1210.

[0256] The speaker, also called a "horn", is used to convert audio electrical signals into sound signals. The electronic device 1200 can listen to music or listen to hands-free calls through the speaker.

[0257] The receiver, also called a handset, is used to convert audio electrical signals into sound signals. When the electronic device 1200 receives a call or a voice message, the user can place the receiver close to the ear to listen to the voice.

[0258] A microphone, also known as a "microphone" or "microphone", is used to convert sound signals into audio electrical signals. When making a call or sending a voice message, the user can put their mouth close to the microphone and speak to input the sound signal into the microphone. The electronic device 1200 can be provided with at least one microphone. In some embodiments, the electronic device 1200 can be provided with two microphones, which can not only collect sound signals but also realize noise reduction function. In some embodiments, the electronic device 1200 can also be provided with three, four or more microphones to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0259] The headphone jack is used to connect wired headphones. The headphone jack can be a USB port, a 3.5mm Open Mobile Terminal Platform (OMTP) standard port, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard port.

[0260] Keys 1290 include a power button, a volume button, and the like. Keys 1290 may be mechanical keys or touch-sensitive keys. Electronic device 1200 receives input from keys 1290 and generates key signal input related to user settings and function control of electronic device 1200.

[0261] Motor 1291 can generate vibration prompts. Motor 1291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations acting on different areas of the display screen 1294 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0262] Indicator 1292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0263] SIM card module 1295 is used to implement the communication functions of the SIM card. SIM card module 1295 may include a SIM card interface, SIM card circuitry, and related auxiliary components. A SIM card can be connected to and disconnected from electronic device 1200 by inserting or removing it from the SIM card interface. Electronic device 1200 may support one or N SIM card interfaces, where N is a positive integer greater than 1. SIM card module 1295 may support Nano SIM cards, Micro SIM cards, SIM cards, and other types. The same SIM card interface can simultaneously accept multiple cards. The multiple cards can be of the same or different types. The SIM card interface can also be compatible with different types of SIM cards. The SIM card interface can also be compatible with external memory cards. Electronic device 1200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, electronic device 1200 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in electronic device 1200 and cannot be separated from it.

[0264] The image processing methods in the aforementioned embodiments can all be implemented in the electronic device 1200 having the aforementioned hardware structure.

[0265] On the basis of the above embodiments, an embodiment of the present application further provides an image processing device, which includes a processor, and the processor is used to execute the image processing method provided by the above embodiments.

[0266] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the image processing method provided in the above embodiment.

[0267] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image processing method provided in the above embodiment.

[0268] The specific implementation methods of the electronic device, computer-readable storage medium, and computer program product containing instructions provided in the embodiments of the present application and the technical effects brought about by them can be found in the specific implementation process of the image processing method provided in the aforementioned embodiments and the technical effects brought about by them, which will not be repeated here.

[0269] In some embodiments, through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0270] The functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0271] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0272] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: Applicable to an electronic device, wherein the camera of the electronic device includes a main camera lens and a camera sensor matched to the main camera lens for imaging; the pixel specification of the camera sensor is a first size, the pixel specification of the image supported by the electronic device for display is a second size, and the first size is larger than the second size; The camera sensor of the electronic device has the ability to output an image in full size at a first zoom factor to an image processor of the electronic device; The image processing method comprises: Displaying a shooting preview interface; the shooting preview interface includes a preview image corresponding to the target zoom ratio and the framing range of the main camera lens; In response to a photographing operation, controlling the camera sensor to output a first image of the first size and a second image of the second size to the image processor, respectively; wherein the first image is an image output by the camera sensor in a full-size output mode, and the second image is an image output by the camera sensor in a combined-size output mode according to the target zoom ratio, and the clarity of the first image is higher than that of the second image; The image processor fuses pixel features of a first pixel area in the first image into a third pixel area of ​​the second image to obtain a third image; the first pixel area is a pixel area corresponding to a target feature object in the first image, and the third pixel area is a pixel area corresponding to the target feature object in the second image; the third image is an image obtained in response to the photographing operation, a zoom ratio of the third image is the target zoom ratio, a size of the third image is the second size, and a clarity of the pixel area corresponding to the target feature object in the third image is higher than a clarity of the third pixel area in the second image.

2. The image processing method according to claim 1, wherein: The step of fusing the first image and the second image to obtain a third image includes: Cropping the first image to obtain a fourth image; wherein the size of the fourth image is a third size, and the third size is smaller than or equal to the second size; The second image and the fourth image are fused to obtain the third image.

3. The image processing method according to claim 2, wherein: The step of cropping the first image to obtain a fourth image includes: Acquire a real-time coordinate range of a target feature object in a preview image; the target feature object is a predetermined object, and the target feature object is located within the framing range of the main camera lens; Determining a first pixel region corresponding to the target feature object in the first image; wherein the first pixel region is a pixel region determined by mapping a real-time coordinate range of the target feature object in the preview image to the first image based on a pixel mapping relationship between the preview image and the first image; A block including the first pixel area is cropped from the first image to obtain the fourth image; wherein the fourth image includes the second pixel area corresponding to the target feature object; and the first pixel area and the second pixel area have the same clarity.

4. The image processing method according to claim 3, wherein: The step of clipping a block containing the first pixel region from the first image to obtain the fourth image includes: determining a target cropping frame based on a central pixel point of the first pixel region, wherein the central point of the target cropping frame coincides with the central pixel point of the first pixel region, the size of the target cropping frame is the second size, and the pixel region covered by the target cropping frame at least partially coincides with the first pixel region; The first image is cropped according to the target cropping frame to obtain the fourth image.

5. The image processing method according to claim 3 or 4, characterized in that: The second image includes a third pixel region corresponding to the target feature object, and the third image includes a fourth pixel region corresponding to the target feature object; the clarity of the fourth pixel region is higher than that of the third pixel region, and the clarity of the second pixel region is the same as that of the fourth pixel region; and / or, The second image further includes a fifth pixel area except the third pixel area, the third image further includes a sixth pixel area except the fourth pixel area, and the definition of the fifth pixel area is the same as that of the sixth pixel area.

6. The image processing method according to claim 5, characterized in that The step of fusing the second image and the fourth image to obtain the third image includes: The pixel features of the second pixel area in the fourth image are superimposed on the third pixel area in the second image to obtain the third image.

7. The image processing method according to claim 1, wherein: The target feature object includes at least one of a human face feature object, an animal face feature object, and a plant or flower feature object.

8. The image processing method according to any one of claims 1 to 4, 6 and 7, characterized in that: Before the step of controlling the camera sensor to output the first image of the first size and the second image of the second size to the image processor in response to a photographing operation, the image processing method further includes: determining whether the target zoom ratio is the first zoom ratio; If the target zoom ratio is the first zoom ratio, the image processing method further includes: In response to the photographing operation, controlling the camera sensor to output the first image of the first size to the image processor; Cropping the first image to obtain a fifth image; wherein the size of the fifth image is the second size; If the target zoom ratio is not the first zoom ratio, the electronic device controls the camera sensor to output a first image of the first size and a second image of the second size to the image processor in response to the photographing operation.

9. The image processing method according to claim 1, wherein: The pixel specification of the camera sensor is 108M, the pixel specification of the image supported by the electronic device for display is 12M, and the first zoom ratio is 3x.

10. The image processing method according to claim 1, wherein: The pixel specification of the camera sensor is 50M, the pixel specification of the image supported by the electronic device for display is 12.5M, and the first zoom ratio is 2x; or, The pixel specification of the camera sensor is 200M, the pixel specification of the image supported by the electronic device for display is 12.5M, and the first zoom ratio is 4x.

11. The image processing method according to any one of claims 1 to 4, 6, 7, 9 and 10, characterized in that: Before the step of controlling the camera sensor to output the first image of the first size and the second image of the second size to the image processor in response to a photographing operation, the image processing method further includes: Detect whether the current camera mode is low-light shooting mode or night scene shooting mode; If the current photographing environment is the low-light photographing mode or the night scene photographing mode, controlling the camera sensor to output the second image of the second size to the image processor according to the target zoom ratio; displaying the second image; If the target zoom ratio is not the low-light shooting mode or the night scene shooting mode, the electronic device performs an operation of controlling the camera sensor to output a first image of the first size and a second image of the second size to the image processor in response to the photographing operation.

12. An electronic device, characterized in that: The electronic device includes a camera, a memory, and a processor, wherein the camera and the memory are coupled to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the image processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the image processing method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 11.

Images