Image processing method, electronic device, computer program product and storage medium

Abstract

The invention provides an image processing method, electronic equipment, a computer program product and a storage medium, and relates to the technical field of terminals. A camera of an electronic device includes a main lens and a camera sensor. And the electronic equipment responds to the photographing operation and executes two picture output operations based on the target zoom ratio. The camera sensor outputs a first image of a first size in a full-size image output manner, and outputs a second image of a second size in a combined-size image output manner. The definition of the first image is higher than that of the second image. A camera sensor of the electronic equipment outputs two frames of images with different sizes to an image processor, and the image processor generates a frame of third image according to the two frames of images with different sizes to serve as a third image finally obtained in response to photographing operation of a user. Thus, the electronic equipment does not need to be provided with a telephoto lens, and images which simultaneously comprise wider backgrounds and are clearer can be obtained only through the main camera.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to an image processing method, an electronic device, a computer program product, and a storage medium. Background Art

[0002] Mobile phones are equipped with cameras, and the function of taking pictures with mobile phone cameras is increasingly optimized. Cameras provide multiple zoom ratios so that mobile phones can capture images at different zoom ratios. The larger the zoom ratio, that is, the larger the focal length, the smaller the camera's field of view, the smaller the framing range, and the fewer photographic objects are included in the framing range. Conversely, the smaller the zoom ratio, that is, the smaller the focal length, the larger the camera's field of view, the larger the framing range, and the more photographic objects are included in the framing range.

[0003] The camera of the mobile phone captures images with a smaller zoom ratio. The larger the field of view, the wider the background is, but the image clarity is relatively lower. In order to obtain a clearer image, the camera of the mobile phone will choose to switch to a telephoto lens. The telephoto lens has a larger focal length, and the captured image is clearer, but the background is smaller.

[0004] The existing image processing solution is to use the main camera lens to capture an image with a larger viewing range, and then use the telephoto lens to capture an image corresponding to the central area of ​​the viewing range, and fuse the two frames of images to obtain a frame of image with a larger viewing range and a clear central area.

[0005] The images captured by the existing image processing solutions rely on the electronic device being equipped with both a main camera lens and a telephoto lens. When the electronic device does not have a telephoto lens, it is impossible to capture images with a large field of view and clear areas in some areas. Summary of the invention

[0006] The embodiments of the present application provide an image processing method, an electronic device, a computer program product, and a storage medium, which are used to solve the technical problem that an electronic device cannot capture an image with a large field of view and clear areas in some areas when it does not have a telephoto lens.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, an image processing method is provided, and the image processing method is applied to an electronic device. The electronic device includes a camera and an image processor, the camera collects images, and the image processor processes the images. The camera includes a main camera lens and a camera sensor, the main camera lens collects light within the viewing range and converges it to the camera sensor, and the camera sensor performs photoelectric conversion to generate an image.

[0009] The pixel size of the camera sensor is a first size, the pixel size of the image supported by the electronic device is a second size, and the first size is larger than the second size. For example, the pixel size of the camera sensor of a commonly used main camera lens is 9000*12000, that is, 108M, and the pixel size of the displayed image is 3000*4000, that is, 12M.

[0010] The image output modes of the camera sensor mainly include full-size image output and merged-size image output. Existing electronic devices use different image output modes to output images to the image processor at different zoom ratios.

[0011] Among them, full-size image output means that the camera sensor directly reads the charge of a single pixel to obtain a corresponding pixel data, so that the pixel specifications of the output image completely correspond to the pixel specifications. The camera sensor outputs a first-size image to the image processor in full-size output at a fixed first zoom ratio. For example, a 108M camera sensor supports a first zoom ratio of 3x for full-size image output. Camera sensors with different pixel specifications may support different first zoom ratios for full-size image output.

[0012] The camera sensor's merged size output method means that the camera sensor merges the charges of adjacent pixels into a corresponding pixel data, and reads out the charges of all pixels after multiple merging to obtain the corresponding image. The image output by the camera sensor in the merged size output method has a smaller image size because it contains fewer pixels. Since the image output by the merged size output method is actually equivalent to downsampling the full-size image, the clarity of the image output by the merged size output method is also lower than that of the image output by the full-size output method.

[0013] The existing photographing scheme is that if the zoom ratio selected by the user is the first zoom ratio, the camera sensor outputs a clear image of the first size to the image processor in the form of full-size output, and the image processor crops the central pixel area of ​​the clear image of the first size to obtain a clear image of the second size. If the zoom ratio selected by the user is not the first zoom ratio, the camera sensor outputs a relatively blurred image of the second size after the merge processing in the form of merged size output. Regardless of the zoom ratio, the camera sensor only outputs one frame of image to the image processor.

[0014] The image processing method provided in this embodiment enables the electronic device to reuse the full-size image output capability of the electronic device at multiple different zoom factors.

[0015] Specifically, the electronic device may display a shooting preview interface, which may include a preview image and a target zoom ratio selected by the user. The electronic device is installed with a camera application, which runs multiple shooting modes, such as photo, portrait, panorama, professional, video and other shooting modes, and the shooting preview interface may be a shooting preview interface in any of the shooting modes.

[0016] If the electronic device does not receive a click operation on the zoom control, the default zoom ratio is 1x, that is, the target zoom ratio is 1x. If the electronic device receives a click operation on the zoom control, the electronic device confirms the target zoom ratio selected by the user according to the click operation on the zoom control.

[0017] When the electronic device is updating and displaying a preview image on a shooting preview interface, if it receives a photo-taking operation from a user, the electronic device performs image acquisition and processing operations in response to the photo-taking operation.

[0018] In response to a photo-taking operation, the electronic device captures an image based on a target zoom ratio. In the process of reading the electrical signal converted by each pixel and obtaining pixel data for image output, the camera sensor performs two image output operations, and the sizes of the images output by the two image output operations are different.

[0019] The first image output operation performed by the camera sensor is to output a first image of a first size in a full-size output mode. The second image output operation performed by the camera sensor is to output a second image of a second size in a combined size output mode. That is, when the electronic device captures an image at a target zoom ratio, different from the existing solution of outputting one frame of image to the image processor at one zoom ratio, the camera sensor in this embodiment outputs two frames of images of different sizes to the image processor in two different image output modes. The clarity of the first image is higher than that of the second image.

[0020] The camera sensor of the electronic device outputs two frames of images of different sizes to the image processor, and the image processor generates a third image frame based on the two frames of images of different sizes as the third image finally obtained in response to the user's photo operation. The zoom ratio of the third image generated by the electronic device is the target zoom ratio.

[0021] The image processing method provided in this embodiment is that when the electronic device captures an image at a target zoom ratio, the electronic device reuses the function of the camera sensor to output a full-size image at a first zoom ratio, and controls the camera sensor to output a first image of the first size in full-size output, and outputs a second image of the second size in a combined size output. The image processor of the electronic device obtains a new image frame based on the first image and the second image as the image corresponding to the current target zoom ratio. The image obtained in this way can include both a wider background and a clearer partial pixel area. In this way, the electronic device does not need to be equipped with a telephoto lens, and can obtain an image that includes both a wider background and a clearer image only through the main camera lens.

[0022] In a possible implementation manner of the first aspect, a solution for an image processor of the electronic device to fuse the first image and the second image to generate a third image is further limited.

[0023] Specifically, the image processor obtains a first image output by the camera sensor in a full-size output mode, and the first image includes a wider background and a clearer foreground feature object or other feature object, and this part of the foreground feature object or other feature object may be the feature object that the user focuses on. The electronic device crops the first image to obtain a fourth image. The size of the fourth image cropped by the electronic device is recorded as the third size, and the third size is less than or equal to the second size. Since the fourth image is directly cropped from the first image, the clarity of the first image is consistent with the clarity of the fourth image, and both are higher than the clarity of the second image output by the combined output mode.

[0024] The image processor fuses the fourth image obtained by cropping the first image with the second image to obtain a third image. In this way, the obtained third image includes both the wider background in the second image and the higher definition pixel area included in the fourth image.

[0025] In a possible implementation of the first aspect, a scheme for the image processor to crop the first image to obtain the fourth image is further limited. In this implementation, the pixel area where the target feature object of the user's attention is located is used as the region of interest, and the clarity of the region of interest in the acquired third image is improved as much as possible. For example, the target feature object can be a human face, an animal face or a flower, or other feature objects of the user's attention. Of course, the electronic device can also provide a custom editing control to receive the identification information of the target feature object input by the user, and improve the clarity of the pixel area where the target feature object of this type is located.

[0026] The image processor obtains the real-time coordinate range of the target feature object in the preview image, where the target feature object is a predetermined object and is located within the framing range of the main camera lens. For example, the image processor may obtain the real-time coordinate range of a face in the preview image of the preview stream.

[0027] The image processor maps the real-time coordinate range of the target feature object in the preview image to the pixel area determined in the first image according to the pixel mapping relationship between the preview image and the first image, and determines the first pixel area corresponding to the target feature object in the first image.

[0028] The image processor cuts out a block including the first pixel area from the first image according to the determined first pixel area of ​​the target feature object to obtain a fourth image. Since the fourth image is obtained by direct cutting, the fourth image includes the second pixel area corresponding to the target feature object; the clarity of the first pixel area is the same as that of the second pixel area.

[0029] In a possible implementation of the first aspect, the step of obtaining a fourth image by cutting out a block including a first pixel area from the first image by the image processor is further limited. The image processor determines a target cropping frame based on a central pixel point of the first pixel area. The center point of the target cropping frame coincides with the center pixel point of the first pixel area, and the size of the target cropping frame is a second size, and the pixel area covered by the target cropping frame coincides with the first pixel area at least partially, or the pixel area covered by the target cropping frame may also completely include the first pixel area. The image processor crops the first image according to the target cropping frame to obtain a fourth image.

[0030] In a possible implementation manner of the first aspect, the clarity of corresponding pixel areas in the images before and after the fusion is further limited.

[0031] The definition scheme for the clarity of the pixel area corresponding to the target feature object in the image includes: the second image includes a third pixel area corresponding to the target feature object, and the third image includes a fourth pixel area corresponding to the target feature object. The clarity of the second pixel area is higher than the clarity of the second pixel area, and the clarity of the second pixel area is the same as the clarity of the fourth pixel area.

[0032] The definition scheme for the clarity of other pixel areas in the image except the pixel area corresponding to the target feature object includes: the second image also includes a fifth pixel area except the third pixel area, the third image also includes a sixth pixel area except the fourth pixel area, and the clarity of the fifth pixel area is the same as the clarity of the sixth pixel area.

[0033] In a possible implementation manner of the first aspect, the step of the image processor fusing the second image and the fourth image to obtain the third image is further limited.

[0034] The image processor superimposes the pixel features of the second pixel area in the fourth image onto the third pixel area in the second image to obtain a third image. The image processor may call an image fusion algorithm or a trained image processing model to perform image fusion processing.

[0035] In a possible implementation manner of the first aspect, the electronic device is limited to select different execution schemes according to a relationship between the target zoom magnification and the first zoom magnification.

[0036] The electronic device determines whether the target zoom ratio is the first zoom ratio. If the target zoom ratio is the first zoom ratio, the electronic device controls the camera sensor to output a first image of a first size to the image processor in response to the photo taking operation, and crops the first image to obtain a fifth image.

[0037] If the target zoom ratio is not the first zoom ratio, the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to the image processor in response to the photographing operation. The electronic device generates a third image based on the first image and the second image.

[0038] The camera sensor of the electronic device has the ability to output full-size images at a first zoom ratio. In order to save computing resources, the electronic device can distinguish between the cases where the target zoom ratio is the first zoom ratio and the case where the target zoom ratio is not the first zoom ratio, and perform different image processing operations respectively.

[0039] In a possible implementation manner of the first aspect, a pixel specification of the camera sensor is 108M, a pixel specification of an image supported for display by the electronic device is 12M, and the first zoom ratio is 3x.

[0040] In a possible implementation manner of the first aspect, a pixel specification of the camera sensor is 50M, a pixel specification of an image supported for display by the electronic device is 12.5M, and the first zoom ratio is 2x;

[0041] or,

[0042] The pixel specification of the camera sensor is 200M, the pixel specification of the image supported by the electronic device for display is 12.5M, and the first zoom ratio is 4x.

[0043] In a possible implementation of the first aspect, whether the shooting environment in which the electronic device performs a photo-taking operation is a high-brightness environment may be further limited. When the shooting environment is a high-brightness environment, the camera sensor outputs the first image in a full-size output mode, and the image signal-to-noise ratio is high and the image quality is good. When the shooting environment is a dark light environment, the camera sensor outputs the first image in a full-size output mode, and the image signal-to-noise ratio is low, the image quality is poor, and the quality of the third image obtained based on the poor first image is also poor.

[0044] When the camera application is turned on, the electronic device can call the ambient light sensor to collect the ambient light brightness of the shooting environment. If the ambient light brightness meets the brightness requirement of the high-brightness environment, the image processing method provided in this embodiment can be executed to obtain a high-quality image. If the ambient light brightness does not meet the brightness requirement of the high-brightness environment, the image processing method provided in this embodiment may not be executed.

[0045] The electronic device detects whether the current shooting mode is a low-light shooting mode or a night scene shooting mode. If the current shooting environment is a low-light shooting mode or a night scene shooting mode, the electronic device controls the camera sensor to output a second image of a second size to the image processor according to a target zoom ratio, and displays the second image. If the target zoom ratio is not a low-light shooting mode or a night scene shooting mode, the electronic device executes an operation of controlling the camera sensor to output a first image of a first size and a second image of a second size to the image processor in response to a shooting operation.

[0046] In a second aspect, the present application provides an electronic device including a camera, a memory and a processor, wherein the camera and the memory are coupled to the processor;

[0047] Memory stores computer-executable instructions;

[0048] The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the image processing method as described in any one of the first aspects.

[0049] In a third aspect, an electronic device is provided, which has the function of implementing the image processing method of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0050] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute any one of the image processing methods in the first aspect.

[0051] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any image processing method in the first aspect.

[0052] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a shooting preview interface of an electronic device;

[0054] Figure 2 A schematic diagram of the pixel arrangement and full-size output of a camera sensor for an electronic device;

[0055] Figure 3 A schematic diagram for plotting the combined dimensions of a camera sensor of an electronic device;

[0056] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the interface and image involved in the image processing method provided in the embodiment of the present application;

[0058] Figure 6 A schematic diagram of merging and outputting images at a 2x magnification involved in the image processing method provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of cropping a first image to obtain a fourth image involved in the image processing method provided in an embodiment of the present application;

[0060] Figure 8 Another schematic diagram of cropping a first image to obtain a fourth image according to the image processing method provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the process of the image processing method provided in an embodiment of the present application;

[0062] Fig.10 A schematic diagram of a processing flow for different target zoom ratios of an image processing method provided in an embodiment of the present application;

[0063] Fig.11 A software framework and flow chart of the image processing method provided in the embodiment of the present application;

[0064] Fig.12 A hardware schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0066] To facilitate understanding, some technical common sense involved in the embodiments of the present application is first introduced.

[0067] The electronic device is equipped with a camera, and the electronic device can receive the user's photo operation and control the camera to execute the corresponding photo process. The camera of the electronic device mainly includes a shutter, a lens module, a camera sensor (Sensor), a camera driver and an image processor, which cooperate to realize the photo function. Among them, the lens module includes multiple groups of lenses, and the multiple groups of lenses are used as lenses with different focal lengths in different combinations. After the shutter is opened, the light enters the lens module, and the lens converges the light onto the camera sensor. The camera sensor converts the collected light signal into an electrical signal, and the electrical signal is transmitted to the image processor for processing to generate an image for display. The camera driver is used to drive the lens module to adjust the focal length and drive the camera sensor to transmit the generated electrical signal to the image processor. The image processor (Graphics Processing Unit, GPU) mentioned here can be a separate chip for performing image processing operations, also known as a display core, a visual processor, a display chip, etc. It is a microprocessor that specializes in image and graphics related operations on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.). Alternatively, the image processor may also be a functional module in a central processing unit (CPU) of an electronic device or other processor that performs image processing operations, without limitation. The image processor may call an algorithm or neural network stored in the electronic device to perform image processing operations.

[0068] The lens module of an electronic device can provide lenses with different focal lengths (or focal lengths) to capture images, which is called zoom. By adjusting the zoom parameters such as the zoom ratio (Zoom Ratio) of the zoom to adjust the field of view (FOV) of the shooting interface, the technical effect of enlarging or reducing the subject in the shooting interface is achieved. The zoom ratios of images captured by electronic devices usually include: 1x, 2x, 3x, 6x, etc. The zoom ratios supported by different electronic devices may be different. Most electronic devices include 1x, 2x, and 3x.

[0069] like Figure 1As shown in (1), it is the shooting preview interface (such as Figure 1 The schematic diagram of U1) shown in (1) in FIG. The shooting preview interface mainly includes: a parameter control area 101, an image display area 102, a mode control area 103, a front and rear camera flip control 104, a gallery control 105, a shutter control 106 and a zoom control 107. Among them, the parameter control area 101 includes a plurality of parameter controls, each of which is used to respond to a shooting parameter adjustment operation input by a user. The parameter controls included in the parameter control area may include, but are not limited to: a flash control, an AI recognition switch control, a color standard control, and a more detailed camera setting control. The image display area 102 can be used to display a preview image, which is an image captured by the electronic device in real time through a camera. The electronic device can refresh the display content in the image display area 102 in real time to facilitate the user to preview the image currently captured by the camera. The mode control area 103 may include a plurality of mode controls corresponding to different shooting modes, such as a photo mode control, a portrait mode control, a video mode control, a professional mode control, an aperture mode control, a night scene mode control and more mode controls. Each mode control can be marked only by text information, for example, "aperture", "night scene", "portrait", "photo", "video", "professional", "more", or it can be displayed by an icon, or a combination of text information and icons.

[0070] The zoom control 107 is used to respond to the zoom operation of the user on the electronic device to adjust the field of view of the lens module, and the image displayed in the image display area 102 changes with the adjustment of the field of view of the lens module. Figure 1 As shown in (1), "1x" is displayed on the zoom control 107, indicating that the current zoom ratio of the electronic device is 1x, and the subsequent x represents the zoom ratio. The user can apply a zoom operation to adjust the zoom ratio by applying a click operation or a sliding operation on the zoom control 107. Of course, the zoom operation can also be applied by a touch operation that does not act on the zoom control 107. For example, the electronic device can pre-define the operation of sliding fingers toward each other or sliding fingers away from each other on the shooting preview interface as a zoom operation, or pre-define the operation of clockwise rotation sliding and counterclockwise rotation sliding on the shooting interface as zoom operations. Such other pre-defined touch operations act on the shooting interface, but not necessarily on the zoom control 107.

[0071] Of course, in other cases, the zoom operation received by the electronic device may also be a zoom operation that does not need to act on the shooting preview interface, but an adjustment operation on a physical device on the surface or side of the electronic device. For example, a physical button or knob is set on the side of the electronic device as a zoom switch, and this part of the physical buttons or knobs can be associated with indicating a specific zoom value selection or zoom ratio adjustment operation, etc. This part of the physical buttons or knobs can be a separate physical device dedicated to receiving zoom operations, or it can be a reused existing volume adjustment button or channel adjustment knob in a shooting scene, etc., without limitation. This situation may be more suitable for scenarios where the electronic device is a retro-styled mobile phone, an elderly phone, a card machine, etc.

[0072] After receiving the zoom operation applied by the user, the electronic device can respond to the zoom operation, determine the target zoom ratio indicated by the zoom operation, and the camera sensor outputs the original image corresponding to the target zoom ratio to the image processor, and the image processor generates an image corresponding to the target zoom ratio based on the original image provided by the camera sensor and sends it for display. The following will specifically explain the principle of the electronic device obtaining images with different zoom ratios through the camera sensor and the image sensor.

[0073] The camera sensor uses the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. Camera sensors are mainly divided into Charged Coupled Device (CCD) and Complementary Metal Oxide Semiconductor (CMOS) according to the device type. Among them, CCD is a semiconductor chip used to capture images. It reads pixel data by charge transfer and has high sensitivity and signal-to-noise ratio. CMOS is a large-scale integrated circuit chip that reads pixel data by line-by-line scanning. The camera sensor of an electronic device can be a CCD sensor or a CMOS sensor.

[0074] The camera sensor includes multiple pixels arranged in an array. The multiple pixels divide the overall light image converged on the light receiving surface of the camera sensor through the lens into unit light images, and each pixel converts the corresponding unit light image into a unit image, and each unit image corresponds to each pixel in the image. In other words, in the full-size image output at the original size of the camera sensor, the pixels correspond to each other one by one.

[0075] like Figure 2(1) in FIG. 1 is a schematic diagram of the pixel arrangement of a camera sensor installed in an electronic device. In this example, the pixel arrangement size of the camera sensor is 9000*12000, 9000*12000=108000000, and the pixel specification can be recorded as 108M (M stands for Millon). For simple illustration, Figure 2 In (1), every 1000 pixels are simplified to 1 cell. The electronic device is equipped with a camera sensor with a 108M pixel specification, and the pixel specification of the generated image is usually 12M, that is, the pixel arrangement size is 3000*4000. It should be noted that the pixel specifications and pixel specifications mentioned here are usually the specifications of the main camera lens of the electronic device's camera and the camera sensor that matches the main camera lens.

[0076] Based on the pixel specifications of the camera sensor and the pixel specifications of the generated image, the electronic device can mainly divide the camera sensor's image output into two types: the first image output method is full-size image output, and the second image output method is merged size image output. When the electronic device collects images at different zoom ratios, the image output method used may also be different.

[0077] In the first output method, the camera sensor of the electronic device outputs the image at full size.

[0078] When the electronic device captures an image at a certain zoom ratio, the camera sensor outputs the image at full size. The zoom ratio corresponding to the electronic device outputting the image at full size is recorded as the first zoom ratio.

[0079] For example Figure 2 As shown in (1), the pixel specification of the image generated by the electronic device is 12M (3000*4000), and the pixel specification of the camera sensor is 108M (9000*12000). When the electronic device captures an image at a 3x zoom ratio, the camera sensor outputs the image in full size, that is, the first zoom ratio is a 3x zoom ratio.

[0080] like Figure 2 As shown in (2), the camera sensor outputs a full-size image to the image processor, and the full-size image output by the camera sensor is recorded as the first image.

[0081] The pixel specification of the image supported by the electronic device for display is 12M, i.e. 3000*4000, while the pixel specification of the full-size image output by the camera sensor received by the image processor is 108M, i.e. 9000*12000. The image processor needs to crop the first image so that the pixel specification of the cropped image is 3000*4000, and the cropped image is recorded as the second image.

[0082] like Figure 2As shown in (3) of FIG. 1 , the image processor of the electronic device generally adopts a cropping scheme of determining the central pixel point of the first image, taking the central pixel point of the first image as the center point, and cropping the first image with a cropping frame of 3000*4000 to obtain the following: Figure 2 The electronic device can display the cropped second image as an image captured by the camera at a 3x magnification.

[0083] The camera sensor of the electronic device outputs a full-size first image, and the second image is directly cropped from the full-size first image without downsampling processing, so the clarity of the displayed image is relatively high.

[0084] The second output method is that the camera sensor of the electronic device outputs the image in a combined size.

[0085] When the electronic device captures an image at a zoom ratio other than the first zoom ratio, the camera sensor reads the pixel data of the pixel array in a binning manner, generates an image of a binning size that meets the pixel specification, and outputs the generated image of the binning size to the image processor. For ease of description, among the multiple zoom ratios supported by the electronic device, the zoom ratios other than the first zoom ratio are recorded as second zoom ratios.

[0086] The way in which the camera sensor of an electronic device merges and reads pixel data refers to that the charges induced in adjacent pixels are added together and read out in a one-pixel mode. Binging is divided into horizontal Binging and vertical Binging. Horizontal Binging adds the charges of adjacent rows together and reads out, while vertical Binging adds the charges of adjacent columns together and reads out. Combining several pixels together to use as one pixel improves the sensitivity of light sensing in dark places and the output speed. When Binging is used for rows and columns at the same time, the aspect ratio of the image does not change, and the resolution of the image is reduced. The merged image output mode mentioned in this embodiment may refer to the use of the merged image output mode for rows and columns at the same time.

[0087] Continue as above Figure 2 In the scenario shown in (1), the pixel size of the image generated by the electronic device is 12M (3000*4000), the pixel size of the camera sensor is 108M (9000*12000), the electronic device supports full-size image output at 3x magnification, and supports merged-size image output at other zoom magnifications except 3x magnification. The second zoom magnification supported by the electronic device for merged-size image output may include 1x, 2x, 4x, 6x, etc.

[0088] like Figure 3 As shown in FIG. 1 , it is a schematic diagram of an electronic device collecting images at a 1x magnification. Figure 3As shown in (1), it is the full-size image converted by the camera sensor, and the camera sensor merges and reads the pixel data. In order to obtain 3000*4000 from a 9000*12000 image, the camera sensor needs to merge the pixel data of three adjacent pixels and read them out as one pixel data. The merging mentioned here can be understood as three-in-one or nine-in-one. The pixel data of three pixels need to be merged into one pixel data to read out horizontally, and the pixel data of three pixels need to be merged into one pixel data to read out vertically.

[0089] like Figure 3 As shown in (2), the 9000*12000 pixel array is divided into three-in-one to obtain pixel regions Z1, Z2...Z11 and Z12, and the pixel regions are merged and read to obtain the corresponding pixel regions A1, A2...A11 and A12, as shown in Figure 3 As shown in (3) in the figure, the image obtained after merging and reading the pixel data is as follows Figure 3 As shown in (4) in the figure. The camera sensor is at 1x magnification. Figure 3 The second image shown in (4) is output to the second image processor. The camera sensor merging and reading method can be convolution or down sampling.

[0090] At other zoom ratios, the camera sensor adaptively adjusts the number of pixels to be read based on the 9000*12000 pixel array, so that the pixel specification of the final output image is 3000*4000. The specific principle can be referred to the principle of merging size output at 1x magnification, which will not be repeated here.

[0091] contrast Figure 2 and Figure 3 , based on the substantially identical or identical framing range, the full-size image captured by the camera sensor is the same, and in response to different zoom ratios, the camera sensor outputs different images to the image processor in different output modes. The image acquired by the electronic device at the first zoom ratio, i.e., 3x, has a higher definition because it is directly cropped from the central area of ​​the full-size image, and the background range is smaller than the framing range. The image acquired by the electronic device at the second zoom ratio has a background range that is consistent with the framing range, but the clarity of the image is reduced due to the merged output.

[0092] In many scenes where electronic devices capture images, it is common that the framing range includes both foreground objects and background objects. Foreground objects can be human faces, animal faces, flowers of plants, etc., and background objects can include landscapes, buildings, etc. Relatively speaking, the area of ​​the foreground object in the framing range will be smaller than the area of ​​the background object in the framing range. In this way, in the image captured by the electronic device, the area of ​​the first pixel area corresponding to the foreground object will be smaller than the area of ​​the second pixel area corresponding to the background object.

[0093] For example Figure 1 As shown in (1) in FIG. 1 , the electronic device captures an image at a common magnification (e.g., 1x), and in the captured image, the first pixel area corresponding to the face object (e.g., Figure 1 Z1 shown in (1) is smaller than the second pixel area corresponding to the background object (such as Figure 1 Z2) shown in (1) in the figure, which can retain more background objects, that is, capture a wider background. Figure 1 As shown in (2) in Figure 1 The image obtained by proportionally cropping the first pixel area shown in (1) shows that the definition of the face object is low, the features such as facial expressions and skin details are few, and the image quality is poor. The image definition that the user actually wants is Figure 1 As shown in (3), the face object has higher definition and more features such as facial expressions and skin details. Such images can be considered as high-quality images.

[0094] In order to obtain a high-quality image that includes a wider background and a clearer face, the electronic device uses a main camera lens and a telephoto lens to capture a frame of image respectively, and then fuses the two frames of image processing. Specifically, the electronic device calls the main camera lens to capture a first image corresponding to the target zoom ratio indicated by the zoom operation, and the first image includes a wider background. The electronic device calls the telephoto lens to capture a second image, and the second image includes a smaller background, but the clarity of the central pixel area is higher. The electronic device fuses the pixel area with higher clarity in the second image into the central pixel area of ​​the first image to obtain a new image, and the new image includes an image of a wider background and a clearer central pixel area. This image processing solution relies on the electronic device being equipped with a telephoto lens. For electronic devices that do not have a telephoto lens, it is impossible to obtain a high-quality image that includes a wider background and a clearer foreground.

[0095] Based on this, this embodiment provides an image processing method, which is applied to an electronic device. The camera of the electronic device may include a main camera lens and a camera sensor that matches the main camera lens for imaging. The pixel arrangement specification of the camera sensor is recorded as a first size, the pixel arrangement specification of the image supported by the electronic device for display is recorded as a second size, and the camera sensor of the electronic device supports full-size image output at a first zoom ratio.

[0096] When the electronic device captures an image at a first zoom ratio, the electronic device controls the camera sensor to output the image at full size. When the electronic device captures an image at a target zoom ratio, the electronic device reuses the function of the camera sensor to output the image at full size at the first zoom ratio, and controls the camera sensor to output the first image at full size and the second image at a combined size, respectively. The image processor of the electronic device obtains a new image frame based on the first image and the second image, which serves as the image corresponding to the current zoom ratio and may include both a wider background and a clearer partial pixel area. In this way, the electronic device does not need to be equipped with a telephoto lens, and can obtain an image that includes both a wider background and a clearer image only through the main camera lens.

[0097] The image processing method provided in this embodiment is applied to electronic devices, and the electronic devices may include personal computers (PCs), tablet computers, laptops, portable computers (such as mobile phones), wearable electronic devices (such as smart watches), augmented reality (AR) and virtual reality (VR) devices, car computers, and other electronic devices with cameras. The following embodiments do not impose any special restrictions on the specific form of the electronic devices.

[0098] like Figure 4 FIG. 1 is a flow chart of an image processing method provided in this embodiment. The provided image processing method mainly includes the following process:

[0099] S41: The electronic device displays a shooting preview interface; the shooting preview interface includes a preview image corresponding to the target zoom ratio and the framing range of the main camera lens.

[0100] The image processing method provided in this embodiment is applied to an electronic device, and the electronic device is installed with a related application program capable of executing the provided image processing method.

[0101] In one example, a third-party image processing application is installed in the electronic device to perform the provided image processing operation. The electronic device displays an icon of the image processing application on the desktop, and when the electronic device receives a click operation on the icon of the image processing application, it starts to perform the image processing operation corresponding to the provided image processing method.

[0102] In another example, the electronic device is installed with a camera application, and the camera application runs multiple shooting modes, such as photo, portrait, panorama, professional, video and other shooting modes. When the camera application runs in one or more shooting modes, for example, when the camera runs in the photo mode, the electronic device starts to execute the provided image processing method. The specific implementation process of the image processing method provided in this embodiment will be explained below by taking the photo mode in the camera application as an example.

[0103] For example Figure 5 As shown in (1) of FIG. 1 , after the electronic device starts the camera application, a shooting preview interface is displayed, and the current shooting mode is the photo mode (such as Figure 5 In the photo mode, the camera lens called by the camera application is the main camera lens, and the main camera lens cooperates with the camera sensor to capture the image within the framing range of the main camera lens, and updates the preview image within the framing range of the main camera lens in the form of a preview stream in real time in the shooting preview interface (such as Figure 5 B) shown in (1) in the figure.

[0104] Figure 5 As shown in (1) in FIG. 1 , the shooting preview interface displayed by the electronic device also includes a zoom control (such as Figure 5 If the electronic device does not receive a click operation on the zoom control, the default zoom ratio is 1x, that is, the target zoom ratio is 1x. If the electronic device receives a click operation on the zoom control, the electronic device confirms the target zoom ratio selected by the user according to the click operation on the zoom control.

[0105] Based on the above background introduction, it can be known that the image captured by the main camera lens and camera sensor of the electronic device is related to the zoom ratio. When the electronic device updates and displays the preview image in the form of a preview stream on the shooting preview interface, if the target zoom ratio changes, the preview image displayed on the shooting preview interface will also change accordingly.

[0106] S42: In response to a photo-taking operation, the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to an image processor respectively; wherein the first image is an image output by the camera sensor in a full-size output mode, and the second image is an image output by the camera sensor in a merged-size output mode according to a target zoom ratio.

[0107] When the electronic device is updating and displaying a preview image on a shooting preview interface, if it receives a photo-taking operation from a user, the electronic device performs image acquisition and processing operations in response to the photo-taking operation.

[0108] In specific implementation, the photo operation can be a click operation of the photo control in the shooting preview interface collected by the electronic device, or the photo operation can also be a voice control operation containing keywords such as "photograph" collected by the electronic device, without limitation.

[0109] like Figure 5 As shown in (2) in FIG. 1 , the electronic device responds to the user applying the camera control (such as Figure 5 The click operation D) shown in (2) opens the shutter, and the light within the viewing range of the main camera lens passes through the open shutter and converges to the light receiving surface of the camera sensor through the main camera lens. Each pixel of the camera sensor senses the light signal and converts it into a corresponding electrical signal.

[0110] In one example, a first size of a pixel arrangement specification within a camera sensor of an electronic device is 9000*12000, a second size of a pixel arrangement specification of an image supported for display by the electronic device is 3000*4000, and the camera sensor of the electronic device supports full-size image output at a first zoom ratio, i.e., 3x.

[0111] In this embodiment, the electronic device captures images based on a target zoom ratio. When the camera sensor reads the electrical signals converted by each pixel and obtains pixel data for image output, it performs two image output operations. The sizes of the images output by the two image output operations are different.

[0112] In a specific implementation, the first image output operation performed by the camera sensor is to output a first image of a first size in a full-size output manner.

[0113] like Figure 5 As shown in (3), the camera sensor reuses the full-size image output function at the target zoom ratio at 3x magnification, and outputs the first image of the first size in the full-size output manner at the target zoom ratio.

[0114] In another specific implementation, the second image output operation performed by the camera sensor is to output a second image of a second size in a merged size output manner.

[0115] like Figure 5 As shown in (3), the camera sensor retains the operation of merging the size to output the image at the target zoom ratio, and reads the pixel data at the target zoom ratio to obtain a second image of a second size.

[0116] That is to say, when the electronic device captures images at a target zoom ratio, different from the existing solution of outputting one frame of image to the image processor at one zoom ratio, the camera sensor in this embodiment outputs two frames of images of different sizes to the image processor in two different output modes.

[0117] In a specific example, the target zoom ratio is 1x, and the camera sensor combines the size to read the pixel data to obtain the specific implementation scheme of the second image as shown in the figure. Figure 3 As shown in (1), it is a full-size 9000*12000 image converted from the camera sensor. Figure 3 As shown in (2), the 9000*12000 pixel array is divided into three-in-one to obtain pixel regions Z1, Z2...Z11 and Z12, and the pixel regions are merged and read to obtain the corresponding pixel regions A1, A2...A11 and A12, as shown in Figure 3 As shown in (3) in FIG. 1 , the second image obtained after merging and reading the pixel data is as follows: Figure 3 As shown in (4) in the figure. The camera sensor is at 1x magnification. Figure 3 The specific implementation method of the camera sensor merging size to output the second image with a 1x magnification can be referred to in the above Figure 3 The images and text descriptions shown will not be repeated here one by one.

[0118] In another specific example, if the target zoom ratio is 2x, the specific implementation scheme of the camera sensor merging size to read pixel data to obtain the second image is as follows: Figure 6 As shown. Figure 6 As shown in (1), it is the full-size image converted by the camera sensor. The camera sensor merges and reads the pixel data. The camera sensor first merges and reads the pixel data in a three-in-one manner, that is, the full-size image P1 of 9000*12000 is Figure 6 As shown in (2) in , the merged size is 3000*4000 image P2. Figure 6 As shown in (3) in , the image P2 is enlarged by 2 times to obtain the image P3, as shown in Figure 6 As shown in (4), the image P3 is cropped from the central area to obtain the image P4. The image P4 obtained after the combined reading, enlargement and cropping is the second image output by the camera sensor at a 2x magnification.

[0119] In another specific example, the target zoom ratio is 3x, and the camera sensor combines the size to read the pixel data to obtain the specific implementation scheme of the second image as follows: Figure 2 As shown. Figure 2 As shown in (2), the camera sensor outputs a full-size image to the image processor.

[0120] The above example takes the pixel size of the camera sensor as 108M, the image size as 12M, and the first zoom ratio supporting full-size output as 3x as an example to explain the specific implementation scheme of the electronic device controlling the camera sensor to output the first image and the second image. In this case, the camera sensor outputs the first image of 108M and the second image of 12M corresponding to the target zoom ratio. At 1x magnification, the camera sensor obtains the second image in a three-in-one combined size output mode.

[0121] In other cases, the pixel size of the camera sensor and the image size may be different.

[0122] For example, in one case, the pixel size of the camera sensor is 50M, and the image size is 12.5M, then the first zoom ratio that the camera sensor supports for full-size output is 2x. At 1x, the camera outputs the first image of 50M in full-size output, and the camera sensor obtains the second image of 12.5M in a two-in-one combined size output.

[0123] For another example, the pixel size of the camera sensor is 200M, and the image size is 12.5M. The first zoom ratio that the camera sensor supports for full-size output is 4x. At a 1x magnification, the camera outputs a 200M first image in full-size output, and the camera sensor obtains a 12.5M second image in a four-in-one combined size output.

[0124] Typically, the pixel specifications and image specifications of a camera sensor in an electronic device are fixed, and the first zoom ratio that supports full-size image output is also fixed. The pixel specifications, image specifications, and first zoom ratios of different electronic devices may be the same or different. In specific implementations, each electronic device may adaptively adjust the execution of the response step according to the pixel specifications and image specifications of the camera sensor, without limitation.

[0125] S43: The electronic device fuses the first image and the second image to obtain a third image; wherein the size of the third image is the second size, and the clarity of at least part of the pixel area of ​​the third image is higher than the clarity of the second image.

[0126] The camera sensor of the electronic device outputs two frames of images of different sizes to the image processor, and the image processor generates a third image frame based on the two frames of images of different sizes as the third image finally obtained in response to the user's photo operation. The zoom ratio of the third image generated by the electronic device is the target zoom ratio.

[0127] When the electronic device updates and displays the shooting preview interface before receiving a shooting operation, it may receive multiple zoom operations applied by the user, and the zoom magnifications indicated by different zoom operations may be different. The electronic device uses the zoom magnification last determined before receiving the shooting operation as the target zoom magnification and generates a third image corresponding to the target zoom magnification.

[0128] In one example, the electronic device may directly display the generated third image on the interface of the camera application or the third-party application. The electronic device may also display a save control on the interface, and the electronic device may save the third image in the album in response to a click operation applied by the user to the save control. The electronic device may also display a cancel control on the interface, and the electronic device may delete the generated third image in response to a click operation applied by the user to the cancel control.

[0129] In another example, the electronic device may also directly save the generated third image to the album. In addition, the electronic device may continue to display the shooting preview interface, and in the album display area within the shooting preview interface, update the image thumbnail previously displayed in the album display area to the thumbnail of the third image. If the electronic device receives a click operation on the album display area within the shooting preview interface, the electronic device may switch the interface to display the third image.

[0130] In a specific implementation, the step of the electronic device generating the third image may further include:

[0131] S431: The electronic device crops the first image to obtain a fourth image; the size of the fourth image is the third size, and the third size is smaller than or equal to the second size.

[0132] S432: The electronic device fuses the second image and the fourth image to obtain a third image.

[0133] The first size of the first image output by the camera sensor is larger than the second size supported by the electronic device for display. The image processor may first perform operations such as cropping or downsampling on the first image to obtain a fourth image of a relatively smaller size. The image processor then generates a third image based on the fourth image and the second image. The third size of the cropped fourth image is less than or equal to the second size.

[0134] There are many specific schemes for the image processor to crop the first image to obtain the fourth image.

[0135] In one example, the image processor crops the pixel region of the fixed coordinates in the first image to obtain the fourth image. For example, the image processor crops the central pixel region of the fixed coordinates in the first image to obtain the fourth image.

[0136] In this case, the image processor directly crops the central pixel area of ​​the first image to obtain the fourth image. The image processor directly crops the central pixel area of ​​the full-size first image to obtain the fourth image. Compared with the central pixel area of ​​the second image obtained by combining the sizes, the feature objects are the same, but the fourth image has higher definition.

[0137] For example Figure 2 As shown in (3), the image processor crops the central pixel area of ​​the first image to obtain Figure 2 The image shown in (4) in .

[0138] In another example, the image processor crops the pixel area with unfixed coordinates in the first image to obtain the fourth image. For example, the image processor may crop according to the pixel area where the target feature object in the first image is located. The coordinates of the pixel area of ​​the target feature object in the first image may not be fixed, and the coordinates of the pixel area cropped in different situations are also not fixed.

[0139] When the image processor crops the first image, it may crop according to the pixel area where the target feature object is located. The target feature object may be a human face object, an animal face object, a flower face object, or other types of feature objects, which are usually the feature objects that the user may focus on. When the image processor crops the first image, the pixel area where the target feature object is located is cropped to obtain the fourth image, so that the pixel area where the feature object that the user focuses on is higher in clarity.

[0140] The electronic device executes the step of S431, which may specifically include:

[0141] S4311: The electronic device determines the real-time coordinate range of the target feature object in the world coordinate system.

[0142] S4312: The electronic device maps the real-time coordinate range to the first image according to the mapping relationship between the world coordinate system and the image coordinate system, and determines a first pixel region where the target feature object is located.

[0143] The electronic device can store a feature recognition algorithm, use the feature recognition algorithm to recognize the target feature object, and determine the real-time coordinate range of the target feature object in the world coordinate system. The following will take the target feature object as a face object as an example to explain the recognition scheme of the target feature object.

[0144] The feature recognition algorithms stored in electronic devices may include feature extraction algorithms and face recognition algorithms. The feature extraction algorithm extracts key features of a face (such as the coordinates, contours, and colors of the eyes, nose, and mouth) for recognition, models the face features using statistical models (such as Gaussian mixture models, principal component analysis, linear discriminant analysis, etc.), and then uses image transformation algorithms to extract features through image transformation (such as integral images, wavelet transforms, etc.).

[0145] The face recognition algorithm may include Eigen Face, Fisher Face and Multi-Task Convolutional Neural Network (MTCNN), etc., which are used for face feature extraction and face detection respectively. The face feature recognition algorithm provided in this embodiment may be other commonly used face recognition algorithms, without limitation.

[0146] Similarly, when the target feature object is a feature object of the type of animal face, flower, etc., the electronic device can also use the feature extraction algorithm and recognition algorithm commonly used in the field to identify the target feature object within the viewing range and determine the real-time coordinate range of the target feature object.

[0147] In one case, the electronic device uses a feature recognition algorithm to detect in real time in a shooting preview scene whether each preview image in a preview stream includes a target feature object. If the target feature object is included, the coordinate range of the target feature object is determined and the coordinate range of the target feature object is fed back to the image processor.

[0148] In this way, after obtaining the first image output by the camera sensor, the image processor can map the real-time coordinate range into the first image according to the mapping relationship between the world coordinate system and the image coordinate system, and determine the first pixel area where the target feature object is located.

[0149] When a photo-taking application is running, the electronic device can continuously suspend the face recognition algorithm so as to identify the face object and detect the area where the face object is located in real time during the photo-taking process.

[0150] In other cases, the electronic device is loaded with a feature recognition algorithm, and uses the feature recognition algorithm to recognize the target feature object in the first image and determine the first pixel area of ​​the target feature object in the first image.

[0151] S4313: The electronic device cuts out a block including the first pixel area from the first image as a fourth image.

[0152] The electronic device determines the first pixel region of the target feature object in the first image based on the above steps, and then crops the first image to obtain a fourth image including the first pixel region.

[0153] In one example, the electronic device may directly perform cropping with the first cropping frame corresponding to the first pixel region, and the pixel features contained in the cropped fourth image are consistent with the pixel features contained in the first pixel region, and the pixel sizes are consistent.

[0154] For example Figure 7 As shown in FIG. 1 , it is a schematic diagram of cropping the first image to obtain the fourth image. Figure 7 (1) in FIG. 1 is a schematic diagram of a first image, and a first cropping frame S1 is determined according to a first pixel region Z1 in the first image. The fourth image is obtained by cropping with the first cropping frame S1, as shown in FIG. Figure 7 As shown in (2) in FIG. 1 , it can be seen that the pixel features included in the fourth image are consistent with the pixel features included in the first pixel area Z1 , and the pixel sizes are consistent.

[0155] In another example, the step of the electronic device cropping the region including the first pixel from the first image may also include the following steps:

[0156] Determine a target cropping frame according to a central pixel point of the first pixel area; wherein the central point of the target cropping frame coincides with the central pixel point of the first pixel area, the size of the target cropping frame is a second size, and the pixel area covered by the target cropping frame coincides with the first pixel area at least partially;

[0157] The fourth image is cropped from the first image according to the target cropping frame.

[0158] In this example, the electronic device may first determine a target cropping frame of the second size according to the first pixel region, and then crop the first image according to the target cropping frame to obtain a fourth image, and the third size of the fourth image is consistent with the second size. The pixel features included in the fourth image may not be completely consistent with the pixel features included in the first pixel region, that is, they at least partially overlap.

[0159] For example Figure 8 As shown in FIG. , it is another schematic diagram of cropping the first image to obtain the fourth image. Figure 8 (1) is a schematic diagram of a first image, and a target cropping frame S2 is determined according to a first pixel area Z1 in the first image. Figure 8 In (1), the pixel area covered by the target cropping frame S2 completely includes the first pixel area.

[0160] The image processor crops the first image according to the target cropping frame S2 to obtain a fourth image, such as Figure 8As shown in (2) in FIG. 1 . It can be seen that the pixel features contained in the fourth image include the pixel features contained in the first pixel region Z1, and the pixel size of the fourth image is consistent with the pixel size of the second image. In this way, when the image processor performs fusion processing on the second image and the fourth image, since the pixel sizes of the two frames of images are consistent, the image processor does not need to perform zero padding and size alignment operations on the fourth image.

[0161] The electronic device obtains a second image and a fourth image based on the above steps, wherein the second image is an image corresponding to the target zoom factor, the second image includes the target feature object and a wider background, and the clarity of the pixel area where the target feature object is located is relatively low. The fourth image includes the target feature object, and the clarity of the pixel area where the target feature object is located is relatively high.

[0162] The electronic device executes S432 to fuse the second image and the fourth image to obtain the third image, which is actually to fuse the pixel features of the target feature object with higher definition included in the fourth image into the second image to enhance the definition of the target feature object in the second image. In this way, in the third image obtained by the fusion process, the definition of the pixel area where the target feature object is located is higher than the definition of the pixel area of ​​the target feature object in the second image.

[0163] There are multiple implementation schemes for the electronic device to execute S432 to fuse the second image and the fourth image to obtain the third image.

[0164] In one example, the electronic device may utilize an image fusion algorithm to fuse pixel features in the fourth image into corresponding pixel regions in the second image to obtain a third image.

[0165] In another example, the electronic device may also load a pre-trained image processing model, and when the electronic device generates the third image, it may also use the image processing model to process the second image and the fourth image to obtain the third image.

[0166] The essence of the model is a trained neural network. A neural network is a network formed by connecting multiple single neural units together. The output of a neural unit can be the input of another or more neural units. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0167] The neural network used to train the image processing model usually includes a deep neural network (DNN) or a convolutional neural network (CNN). Among them, a deep neural network is also called a multi-layer neural network, which can be understood as a neural network with multiple hidden layers. According to the position of different layers, the DNN can be divided into three categories: input layer, hidden layer and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the multiple layers in the middle are hidden layers. The layers can be fully connected. In other words, any neuron in the i-th layer can be connected to any neuron in the i+1-th layer.

[0168] Convolutional neural network is a deep neural network with convolutional structure. Convolutional neural network contains a feature extractor composed of convolution layer and subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units.

[0169] Electronic devices usually need multiple groups of sample images for model training. Each group of sample images includes two types of images, one is the input sample image containing the features to be classified, and the other is the target sample image containing the target features. It should be noted that the features to be classified here refer to the features that need to be processed such as identification, classification, labeling, etc., such as features with clear faces, blurred faces, etc. The target features are one or several types of features that need to be identified or processed, such as features with fused images, clarity, etc. The image content of the input sample image and the target sample image is associated.

[0170] The input sample image is used as the input value of the neural network, and the target sample image is used as the target value of the neural network. The input sample image contains various basic elements that may be included in the image to be processed by the image processing model. The target sample image may be an image obtained by processing the input sample image according to user requirements, and the target sample image may remove elements that the user does not need based on the input sample image. In other words, the element contents of the input sample image and the target sample image are basically the same, but the feature attributes may be different.

[0171] In this embodiment, the electronic device can train a neural network to obtain an image processing model, and then use the image processing model to perform image processing operations. The electronic device can also directly obtain an image processing model trained by other devices to perform the image processing method provided in this embodiment.

[0172] The process of an electronic device training a neural network to obtain an image processing model mainly includes: the electronic device obtains a sample image and a neural network without setting relevant parameters of a weight matrix, inputs the sample image into the neural network for iterative training, and gradually optimizes the weight matrix in the neural network according to the loss function until the loss function converges, so that the weight matrix in the neural network reaches the optimal value. At this time, the neural network can be used as an image processing model.

[0173] The sample image acquired by the electronic device includes an input sample image and a target sample image. The input sample image may include a first sample image and a second sample image, and the pixel specifications of the first sample image and the second sample image are both the second size. The first sample image includes a wider background and a sample feature object, and the clarity of the pixel area where the sample feature object is located in the first sample image is relatively low. The second sample image includes a sample feature object, and the clarity of the pixel area where the sample feature object is located in the second sample image is relatively high.

[0174] The pixel specification of the target sample image is also the second size, the target feature object includes the sample feature object and a wider background, and the pixel area where the sample feature object is located in the target sample image has a higher definition. Compared with the first sample image, the pixel area where the sample feature object is located in the target sample image has a higher definition.

[0175] The electronic device uses the first sample image and the second sample image as input values ​​and the target sample training as the target value to iteratively train the neural network without setting the weight parameter. Specifically, the electronic device uses the loss function to calculate the difference between the predicted value and the true value, and then uses the difference back propagation to modify the weight matrix; then the sample data used for training is input into the neural network after the weight matrix is ​​modified, and multiple iterative training is performed until the difference is minimized, that is, the neural network converges.

[0176] When the neural network uses input values ​​and target values ​​for iterative training, the error back propagation algorithm is used to correct the parameters of the weight matrix in the initial neural network during the training process, so that the reconstruction error loss of the neural network becomes smaller and smaller. Specifically, the difference between the predicted value and the target value obtained in each iterative training is calculated, and then the difference is reversely transferred to the neural network to modify the weight matrix of the neural network, and the neural network after the modified weight matrix is ​​iteratively trained again, and the weight matrix of the neural network is continued to be updated with the difference obtained from the iterative training again. Repeat multiple iterative trainings and reversely modify the weight matrix with the difference of each iterative training until the difference between the predicted value and the target value obtained is within the allowable loss range of the loss function, that is, the loss function converges, so that the convergence of the neural network is achieved through multiple iterative training. The back propagation algorithm is a back propagation movement dominated by error loss, which is used to obtain the parameters of the optimal neural network model. Of course, iterative training can also be iteratively optimized using other similar algorithms without limitation.

[0177] The electronic device trains an image processing model according to the above scheme and stores the trained image processing model. When the image processor performs fusion processing on the second image and the fourth image, the second image and the fourth image can be input into the image processing model for image processing, and the image output by the image processing model is used as the third image.

[0178] like Fig. 9 As shown, it is a process schematic diagram of the image processing method. When the electronic device performs a photo-taking operation at a target zoom ratio, the camera sensor is controlled to output a first image of a first size and a second image of a second size, and the first image of the first size is cropped according to the first pixel area where the target feature object is located to obtain a fourth image. The electronic device inputs the second image and the fourth image into the image processing model to obtain a third image. The third image includes both a wider background corresponding to the framing range of the main camera lens at the target zoom ratio and a clearer target feature object, and the image quality is high.

[0179] In a specific implementation, another image processing method is also provided. The difference between the provided image processing method and the above-mentioned image processing method includes that the electronic device can select different execution schemes according to the relationship between the target zoom ratio and the first zoom ratio.

[0180] Specifically, Fig.10 As shown, the provided image processing method mainly includes the following steps:

[0181] S1001, the electronic device determines whether the target zoom ratio is a first zoom ratio;

[0182] If the target zoom ratio is the first zoom ratio, executing S1002, the electronic device controls the camera sensor to output a first image of a first size to the image processor in response to the photographing operation;

[0183] S1003: The electronic device generates a fifth image; wherein the fifth image is an image obtained by cropping the first image by an image processor.

[0184] If the target zoom ratio is not the first zoom ratio, S1004 is executed, and the electronic device controls the camera sensor to output a first image of a first size and a second image of a second size to the image processor in response to the photographing operation.

[0185] S1005: The electronic device generates a third image according to the first image and the second image.

[0186] The camera sensor of the electronic device has the ability to output full-size images at a first zoom ratio. In order to save computing resources, the electronic device can distinguish between the cases where the target zoom ratio is the first zoom ratio and the case where the target zoom ratio is not the first zoom ratio, and perform different image processing operations respectively.

[0187] In one case, the target zoom ratio is not the first zoom ratio. The electronic device can control the camera sensor to output a first image of a first size and a second image of a second size to the image processor, respectively. The specific implementation process can refer to the aforementioned Figure 4 The specific implementation process of the image processing method shown will not be repeated here one by one.

[0188] In another case, the target zoom ratio is the first zoom ratio, for example, for a 108M camera sensor, the target zoom ratio is 3x.

[0189] In this case, the camera sensor directly outputs the full-size first image to the image processor at the 3x magnification based on the full-size image output capability. The image processor crops the first image, for example Figure 2 The fifth image is obtained by cropping the central pixel area as shown in (3). Since the full-size image has a high definition, the fifth image obtained by cropping also has a high definition, that is, a high-quality image with high definition can be obtained without collecting and fusing multiple frames of images.

[0190] In this embodiment, the electronic device distinguishes between the case where the target zoom ratio is the first zoom ratio and the case where the target zoom ratio is not the first zoom ratio, which can save computing resources and obtain high-quality images.

[0191] In other embodiments, the electronic device may also distinguish the brightness of the current shooting environment and select different processing schemes. When the shooting environment is a high-brightness environment, the camera sensor outputs the first image in a full-size output mode, and the image signal-to-noise ratio is high and the image quality is good. When the shooting environment is a dark light environment, the camera sensor outputs the first image in a full-size output mode, and the image signal-to-noise ratio is low, the image quality is poor, and the quality of the third image obtained based on the poor first image is also poor.

[0192] Based on this, this embodiment may also limit the prerequisite for performing the image processing operation to a high-brightness shooting environment. The details are as follows:

[0193] The electronic device detects whether the current shooting mode is a low-light shooting mode or a night scene shooting mode;

[0194] If the current shooting environment is a low-light shooting mode or a night scene shooting mode, control the camera sensor to output a second image of a second size to the image processor according to the target zoom ratio;

[0195] displaying a second image;

[0196] If the target zoom ratio is not the low-light shooting mode or the night scene shooting mode, the electronic device executes an operation of controlling the camera sensor to output a first image of a first size and a second image of a second size to the image processor in response to the photographing operation.

[0197] In one case, the electronic device may be limited to not executing the image processing operation provided in this embodiment in the night scene mode.

[0198] In another case, the electronic device can also call the ambient light sensor to collect the ambient light brightness of the shooting environment when the camera application is turned on. If the ambient light brightness meets the brightness requirement of the high-brightness environment, the image processing method provided in this embodiment can be executed to obtain a high-quality image. If the ambient light brightness does not meet the brightness requirement of the high-brightness environment, the image processing method provided in this embodiment may not be executed.

[0199] The image processing method provided in the above embodiment can obtain a wider background and a clearer face image based on the main camera lens of the camera. The camera sensor reuses the ability to perform on-chip zoom (Insensor Zoom) at the first zoom ratio when the target zoom ratio is not the first zoom ratio, and expands the position of cropping after full-size image output from the central pixel area to the pixel area guided by the user's focus point, providing 3X high-definition image quality for the user's area of ​​interest. Through the subsequent image fusion capability, the user's more focused position can present higher clarity, realizing zoom fusion based on a single camera, and solving the dependency problem of the telephoto lens.

[0200] The above embodiment explains the specific implementation process of the image processing method from the perspective of the electronic device. The following will explain the specific implementation process of the image processing method from the perspective of the internal software architecture of the electronic device.

[0201] like Fig.11 The following is a detailed explanation of the image processing process of the electronic device in combination with the internal architecture of the electronic device.

[0202] Specifically, the internal architecture of an electronic device can be divided into four layers, from top to bottom, namely, the application layer (APP), the framework layer (FWK), the hardware abstraction layer (HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included without limitation.

[0203] The application layer may include a series of application packages, such as the camera application involved in this embodiment. In addition, the application layer also includes applications such as gallery, image processing applications with camera functions, etc. The application package may also include applications such as call, calendar, map, navigation, music, video, short message, etc.

[0204] The framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The framework layer includes some predefined functions.

[0205] The framework layer runs a camera service, which can be called by the camera application to implement functions related to shooting. In addition, the framework layer can also include a window manager, a content provider, a view system, a resource manager, a notification manager, etc. Among them, the window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to the application. The data can include video, images, audio, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. The notification information may also be, for example, a text message displayed in the status bar, a reminder sound, vibration of the power terminal, flashing indicator light, etc. It should be noted that the camera application may also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and this embodiment does not impose any restrictions on this.

[0206] The kernel layer is a layer between hardware and software. The kernel layer includes at least a camera driver. The camera driver can be used to drive a hardware module with a shooting function, such as a camera sensor. In other words, the camera driver is responsible for data interaction with the camera sensor. The kernel layer may also include a display driver, an audio driver, a sensor driver, etc., which are not limited in this embodiment. In this solution, the electronic device includes a camera, the camera includes a main camera lens, the camera driver is used to drive the main camera lens to open the shutter and adjust the lens focal length of the lens combination, and the camera sensor interacts with the camera driver for data to realize the image acquisition function of the camera.

[0207] The hardware abstraction layer can encapsulate the drivers in the kernel layer and provide a calling interface to the framework layer, shielding the implementation details of the underlying hardware. Fig.11As shown, the hardware abstraction layer may include a camera call processing module (Camera HAL), etc. The camera call processing module is the core software framework of the camera, and the camera call processing module includes an interface module, a sensor call module (Sensor Node), an image processing module, etc. The interface module, the sensor call module, and the image processing module are components in the image data and control instruction transmission pipeline in the camera call processing module. The image processing module can call the image processor to implement the corresponding functions of the aforementioned image processing.

[0208] Specifically, the sensor module may be a control node for the camera sensor, and the sensor module may control the camera sensor through the camera driver. The interface module may be a software interface for the application framework layer, and is used to interact with the application framework layer for data. Of course, the interface module may also interact with the image processing module in the camera call processing module for data.

[0209] The image processing module can process the first image and the second image output by the camera sensor, crop the first image to obtain a fourth image, and fuse the second image and the fourth image to obtain a third image.

[0210] like Fig.11 As shown, at the application layer, the camera application can receive the user's zoom operation and pass the target zoom ratio selected by the user to the camera service of the framework layer.

[0211] At the framework layer, the camera service receives the target zoom ratio and generates a photo request, which indicates the target zoom ratio and the way in which the camera sensor outputs the image: outputting the first image in full size and / or outputting the second image in a merged size.

[0212] The camera service stores the first zoom ratio. If it is determined that the target zoom ratio is not the first zoom ratio, the generated photo request indicates that the camera sensor outputs the image in the following manner: outputting the first image in full size and outputting the second image in a merged size.

[0213] If the camera service determines that the target zoom ratio is the first zoom ratio, the generated photo request instructs the camera sensor to output the image in a manner of outputting the first image in full size.

[0214] The camera service sends the photo request to the hardware abstraction layer through the application program interface.

[0215] At the hardware abstraction layer, the interface module receives the photo request and passes the photo request to the sensor calling module and the image processing module respectively.

[0216] The sensor calling module can call the camera sensor through the camera driver, so that the camera sensor outputs a first image of a first size and a second image of a second size respectively. The camera sensor outputs the first image of the first size in a full-size output manner.

[0217] The camera sensor first downsamples to obtain an image of a second size in a merged size output manner, then magnifies the image of the second size according to the zoom ratio, and then crops the image according to the central pixel area to obtain a second image of the second size.

[0218] The image processing module receives the first image and crops the first image to obtain a fourth image. Specifically, the image processing module first determines a first pixel region in the first image based on the face coordinates, crops the first image with the first pixel region, and obtains a new image as the fourth image.

[0219] The image processing module receives the second image, and fuses the second image and the fourth image to obtain a third image.

[0220] The image processing module uploads the third image (via the interface module) to the framework layer, and the camera service and view system of the framework layer render the third image, and then upload it to the application layer for display or storage.

[0221] Understandably, Fig.11 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0222] In addition, it is understood that in order to implement the image processing method in this embodiment, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0223] In addition, an embodiment of the present application further provides an electronic device, including a camera, a memory, and a processor;

[0224] Memory stores computer-executable instructions;

[0225] The processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the image processing method provided by the above embodiment. In addition to these main components, the electronic device also includes components for realizing basic functions. Fig.12 Provide specific instructions.

[0226] like Fig.12 The figure shows a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of the present application. The electronic device 1200 may include a processor 1210, an external memory interface 1220, an internal memory 1221, a Universal Serial Bus (USB) interface 1230, a charging management module 1240, a power management module 1241, a battery 1242, an antenna 1, an antenna 2, a mobile communication module 1250, a wireless communication module 1260, an audio module 1270, a speaker, a receiver, a microphone, an earphone interface, a sensor module 1280, a button 1290, a motor 1291, an indicator 1292, a camera 1293, a display screen 1294, and a SIM card module 1295. The sensor module 1280 may include a pressure sensor 1280A, a gyroscope sensor 1280B, an ambient light sensor 1280C, a touch sensor 1280D, and the like.

[0227] The structure shown in the embodiment of the present invention does not limit the electronic device 1200. It may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0228] The processor 1210 may include one or more processing units. For example, the processor 1210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0229] The controller can be a decision maker that directs the various components of the electronic device 1200 to work in coordination according to the instructions. It is the nerve center and command center of the electronic device 1200. The controller generates an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0230] The processor 1210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1210 is a high-speed cache memory, which can store instructions or data that the processor 1210 has just used or cyclically used. If the processor 1210 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1210, and thus improves the efficiency of the system.

[0231] In some embodiments, the processor 1210 may include an interface. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0232] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDL) and a serial clock line (SCL). In some embodiments, the processor 1210 may include multiple groups of I2C buses. The processor 1210 may be coupled to the touch sensor 1280D, the charger, the flash, the camera 1293, etc. through different I2C bus interfaces. For example: the processor 1210 may be coupled to the touch sensor 1280D through the I2C interface, so that the processor 1210 communicates with the touch sensor 1280D through the I2C bus interface, thereby realizing the touch function of the electronic device 1200.

[0233] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1210 with the camera 1293, the display 1294, the wireless communication module 1260, the audio module 1270, the sensor module 1280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0234] The USB interface 1230 may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 1230 may be used to connect a charger to charge the electronic device 1200, or may be used to transmit data between the electronic device 1200 and a peripheral device. The USB interface 1230 may also be used to connect headphones to play audio through the headphones. The USB interface 1230 may also be used to connect other electronic devices, such as AR devices, etc.

[0235] The interface connection relationship between the modules shown in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 1200. The electronic device 1200 may use different interface connection methods in the embodiment of the present invention, or a combination of multiple interface connection methods.

[0236] The charging management module 1240 can be a rechargeable battery or a disposable battery. For a rechargeable battery, it can receive charging input through a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1240 can receive charging input from a wired charger through the USB interface 1230. In some wireless charging embodiments, the charging management module 1240 can receive wireless charging input through a wireless charging coil of the electronic device 1200. While the charging management module 1240 is charging the battery 1242, it can also power the electronic device 1200 through the power management module 1241.

[0237] The power management module 1241 is used to connect the battery 1242, the charging management module 1240 and the processor 1210. The power management module 1241 receives input from the battery 1242 and / or the charging management module 1240, and provides power to the processor 1210, the memory 1221, the external memory interface 1220, the display screen 1294, the camera 1293, and the wireless communication module 1260. The power management module 1241 can also be used to monitor parameters such as the capacity of the charging management module, the number of cycles of the charging management module, and the health status (leakage, impedance) of the charging management module. In some embodiments, the power management module 1241 can also be set in the processor 1210. In some embodiments, the power management module 1241 and the battery 1242 can also be set in the same device.

[0238] The wireless communication function of the electronic device 1200 can be implemented through antenna 1, antenna 2, mobile communication module 1250, wireless communication module 1260, modem and baseband processor.

[0239] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 1200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch.

[0240] The mobile communication module 1250 may provide a communication processing module for wireless communication solutions including 2G / 3G / 4G / 5G, etc., applied to the electronic device 1200. The mobile communication module 1250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 1250 receives electromagnetic waves from the antenna 1, and performs filtering, amplification, and other processing on the received electromagnetic waves, and transmits them to the modem for demodulation. The mobile communication module 1250 may also amplify the signal modulated by the modem, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 1250 may be arranged in the processor 1210. In some embodiments, at least some of the functional modules of the mobile communication module 1250 may be arranged in the same device as at least some of the modules of the processor 1210.

[0241] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.), or displays an image or video through a display screen 1294. In some embodiments, the modem may be an independent device. In some embodiments, the modem may be independent of the processor 1210 and be set in the same device as the mobile communication module 1250 or other functional modules.

[0242] The wireless communication module 1260 can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc., which are applied to the electronic device 1200. The wireless communication module 1260 can be one or more devices integrating at least one communication processing module. The wireless communication module 1260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 1210. The wireless communication module 1260 can also receive the signal to be sent from the processor 1210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0243] In some embodiments, the antenna 1 of the electronic device 1200 is coupled to the mobile communication module 1250, and the antenna 2 is coupled to the wireless communication module 1260, so that the electronic device 1200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include Global Positioning System (SatelliteBased Augmentation Systems, SBAS), Global Navigation Satellite System (GLObal NavigAtion Satellite System, GLONASS), BeiDou navigation Satellite system (BeiDou navigation Satellite system, BDS), Quasi-Zenith Satellite System (Quasi-Zenith Satellite System, QZSS) and / or Satellite Based Augmentation System (Satellite Based Augmentation Systems, SBAS).

[0244] The electronic device 1200 implements the display function through a GPU, a display screen 1294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1210 may include one or more GPUs that execute program instructions to generate or change display information.

[0245] The display screen 1294 is used to display images, videos, etc. The display screen 1294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLED, Micro-OLED, quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 1200 may include 1 or N display screens 1294, where N is a positive integer greater than 1.

[0246] The electronic device 1200 can realize the shooting function through ISP, camera 1293, video codec, GPU, display screen and application processor.

[0247] The ISP is used to process the data fed back by the camera 1293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and chromaticity of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 1293.

[0248] The camera 1293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 1200 may include 1 or N cameras 1293, where N is a positive integer greater than 1.

[0249] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 1200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0250] Video codecs are used to compress or decompress digital videos. The electronic device 1200 may support one or more video codecs. Thus, the electronic device 1200 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0251] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 1200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0252] The external memory interface 1220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1200. The external memory card communicates with the processor 1210 through the external memory interface 1220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0253] The internal memory 1221 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 1210 executes various functional applications and data processing of the electronic device 1200 by running the instructions stored in the internal memory 1221. The memory 1221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 1200 (such as audio data, a phone book, etc.), etc. In addition, the memory 1221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash memory (Universal Flash Storage, UFS), etc.

[0254] The electronic device 1200 can implement audio functions such as music playing and recording through the audio module 1270, speakers, receivers, microphones, headphone jacks, and application processors.

[0255] The audio module 1270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 1270 can also be used to encode and decode audio signals. In some embodiments, the audio module 1270 can be arranged in the processor 1210, or some functional modules of the audio module 1270 can be arranged in the processor 1210.

[0256] The speaker, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 1200 can listen to music or listen to a hands-free call through the speaker.

[0257] The receiver, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 1200 receives a call or voice message, the voice can be received by placing the receiver close to the human ear.

[0258] A microphone, also called a "microphone" or "microphone", is used to convert sound signals into audio electrical signals. When making a call or sending a voice message, the user can put their mouth close to the microphone and speak to input the sound signal into the microphone. The electronic device 1200 can be provided with at least one microphone. In some embodiments, the electronic device 1200 can be provided with two microphones, which can not only collect sound signals but also realize noise reduction function. In some embodiments, the electronic device 1200 can also be provided with three, four or more microphones to realize the collection of sound signals, noise reduction, identification of sound sources, and realization of directional recording function, etc.

[0259] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0260] The key 1290 includes a power button, a volume button, etc. The key 1290 may be a mechanical key or a touch key. The electronic device 1200 receives the key 1290 input and generates a key signal input related to the user settings and function control of the electronic device 1200.

[0261] Motor 1291 can generate vibration prompts. Motor 1291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations acting on different areas of the display screen 1294 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0262] Indicator 1292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0263] The SIM card module 1295 is used to implement the communication function of the SIM card. The SIM card module 1295 may include a SIM card interface, a SIM card circuit, and related auxiliary devices. The SIM card can be connected to and separated from the electronic device 1200 by inserting it into the SIM card interface or pulling it out from the SIM card interface. The electronic device 1200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card module 1295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface at the same time. The types of the multiple cards can be the same or different. The SIM card interface can also be compatible with different types of SIM cards. The SIM card interface can also be compatible with external memory cards. The electronic device 1200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 1200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 1200 and cannot be separated from the electronic device 1200.

[0264] The image processing methods in the aforementioned embodiments can all be implemented in the electronic device 1200 having the aforementioned hardware structure.

[0265] On the basis of the above embodiments, an embodiment of the present application further provides an image processing device, the image processing device includes a processor, and the processor is used to execute the image processing method provided by the above embodiments.

[0266] The embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the image processing method provided in the above embodiment.

[0267] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method provided in the above embodiment.

[0268] The specific implementation methods of the electronic device, computer-readable storage medium, and computer program product containing instructions provided in the embodiments of the present application and the technical effects brought about by them can be found in the specific implementation process of the image processing method provided in the aforementioned embodiments and the technical effects brought about by them, which will not be repeated here.

[0269] In some embodiments, through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0270] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0271] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0272] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the camera of the electronic device comprises a main camera lens and a camera sensor matching the main camera lens for imaging; the pixel specification of the camera sensor is a first size, the pixel specification of the image supported by the electronic device for display is a second size, and the first size is larger than the second size; The camera sensor of the electronic device has the ability to output an image to an image processor of the electronic device in a full-size output mode at a first zoom factor; The image processing method comprises: Displaying a shooting preview interface; the shooting preview interface includes a preview image corresponding to the target zoom ratio and the framing range of the main camera lens; In response to a photographing operation, controlling the camera sensor to output a first image of the first size and a second image of the second size to the image processor respectively; wherein the first image is an image output by the camera sensor in a full-size output mode, and the second image is an image output by the camera sensor in a combined-size output mode according to the target zoom ratio, and the definition of the first image is higher than that of the second image; A third image is obtained by fusing the first image and the second image; a size of the third image is the second size, and a clarity of at least a portion of pixel areas of the third image is higher than a clarity of the second image.

2. The image processing method according to claim 1, characterized in that: The step of fusing the first image and the second image to obtain a third image comprises: Cropping the first image to obtain a fourth image; wherein a size of the fourth image is a third size, and the third size is smaller than or equal to the second size; The second image and the fourth image are fused to obtain the third image.

3. The image processing method according to claim 2, characterized in that: The step of cropping the first image to obtain a fourth image comprises: Acquire a real-time coordinate range of a target feature object in a preview image; the target feature object is a predetermined object, and the target feature object is located within the framing range of the main camera lens; Determine a first pixel region corresponding to the target feature object in the first image; wherein the first pixel region is a pixel region determined by mapping the real-time coordinate range of the target feature object in the preview image to the first image according to a pixel mapping relationship between the preview image and the first image; The fourth image is obtained by cropping a block including the first pixel area from the first image; wherein the fourth image includes the second pixel area corresponding to the target feature object; and the first pixel area has the same clarity as the second pixel area.

4. The image processing method according to claim 3, characterized in that: The step of cutting out a block including the first pixel region from the first image to obtain the fourth image comprises: Determining a target cropping frame according to a central pixel point of the first pixel area; wherein the central point of the target cropping frame coincides with the central pixel point of the first pixel area, the size of the target cropping frame is the second size, and the pixel area covered by the target cropping frame coincides with the first pixel area at least partially; The first image is cropped according to the target cropping frame to obtain the fourth image.

5. The image processing method according to claim 3 or 4, characterized in that: The second image includes a third pixel area corresponding to the target feature object, and the third image includes a fourth pixel area corresponding to the target feature object; the definition of the second pixel area is higher than the definition of the second pixel area, and the definition of the second pixel area is the same as the definition of the fourth pixel area; and / or, The second image further includes a fifth pixel region except the third pixel region, the third image further includes a sixth pixel region except the fourth pixel region, and the clarity of the fifth pixel region is the same as that of the sixth pixel region.

6. The image processing method according to claim 5, characterized in that: The step of fusing the second image and the fourth image to obtain the third image comprises: The pixel features of the second pixel area in the fourth image are superimposed on the third pixel area in the second image to obtain the third image.

7. The image processing method according to any one of claims 3 to 6, characterized in that: The target feature object includes at least one of a human face feature object, an animal face feature object, and a plant flower feature object.

8. The image processing method according to any one of claims 1 to 7, characterized in that: Before the step of controlling the camera sensor to output the first image of the first size and the second image of the second size to the image processor in response to the photographing operation, the image processing method further includes: Determining whether the target zoom ratio is the first zoom ratio; If the target zoom ratio is the first zoom ratio, the image processing method further includes: In response to the photographing operation, controlling the camera sensor to output the first image of the first size to the image processor; Cropping the first image to obtain a fifth image; wherein the size of the fifth image is the second size; If the target zoom ratio is not the first zoom ratio, the electronic device executes an operation of controlling the camera sensor to output a first image of the first size and a second image of the second size to the image processor in response to the photographing operation.

9. The image processing method according to any one of claims 1 to 8, characterized in that: The pixel specification of the camera sensor is 108M, the pixel specification of the image displayed by the electronic device is 12M, and the first zoom ratio is 3x.

10. The image processing method according to any one of claims 1 to 8, characterized in that: The pixel specification of the camera sensor is 50M, the pixel specification of the image supported by the electronic device for display is 12.5M, and the first zoom ratio is 2x; or, The pixel specification of the camera sensor is 200M, the pixel specification of the image supported for display by the electronic device is 12.5M, and the first zoom ratio is 4x.

11. The image processing method according to any one of claims 1 to 10, characterized in that: Before the step of controlling the camera sensor to output the first image of the first size and the second image of the second size to the image processor in response to the photographing operation, the image processing method further includes: Detect whether the current shooting mode is low-light shooting mode or night scene shooting mode; If the current photographing environment is the low-light photographing mode or the night scene photographing mode, controlling the camera sensor to output the second image of the second size to the image processor according to the target zoom ratio; displaying the second image; If the target zoom ratio is not the low-light shooting mode or the night scene shooting mode, the electronic device executes an operation of controlling the camera sensor to output a first image of the first size and a second image of the second size to the image processor in response to the photographing operation.

12. An electronic device, characterized in that: The electronic device comprises a camera, a memory and a processor, wherein the camera and the memory are both coupled to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the image processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the image processing method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 11.

Images