Image processing method, electronic device and readable storage medium

Abstract

The present application provides an image processing method, an electronic device, and a readable storage medium, and relates to the field of image processing technology. In the method, when a user uses the shooting function of an electronic device, the user adjusts the first zoom ratio to the second zoom ratio. After the electronic device detects the user operation, it obtains the original image based on the first zoom ratio, and crops the original image according to the second zoom ratio to obtain the first image; and performs image signal processing on the first image to display the processed image. In the method provided by the present application, in a zoom shooting scene, when a change in the zoom ratio is detected, the original image is cropped according to the zoom ratio and then image signal processing is performed, which can reduce the ISP processing time, thereby increasing the shooting speed and improving the user experience. In addition, the memory required to be occupied during the ISP processing process is reduced, which can optimize the memory and thereby improve the overall shooting performance.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an electronic device and a readable storage medium. Background Art

[0002] At present, electronic devices generally support zoom shooting when shooting, wherein image zoom means that in the process of shooting an image by an electronic device, the size of the photographed object in the acquired image can be changed by changing the zoom ratio. However, no matter how many times the user changes the zoom ratio, the sensor will output the original image according to the original ratio, and after processing the original image, the electronic device outputs the target image corresponding to the zoom ratio.

[0003] Based on the above solution, the speed at which the electronic device outputs the target image is often slow, affecting the user experience. Summary of the invention

[0004] The present application provides an image processing method, an electronic device and a readable storage medium, which can improve the shooting speed. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: an electronic device detects a first operation performed by a user when using a shooting function of the electronic device, the first operation being used to adjust a first zoom ratio to a second zoom ratio, the electronic device responds to the first operation, and uses a first camera of the electronic device to acquire an original image based on the first zoom ratio; after acquiring the original image, the electronic device crops the original image according to the second zoom ratio to obtain a first image; and performs image signal processing on the first image; and displays the image after the image signal processing on the first image.

[0006] In an embodiment of the present application, in a zoom shooting scene, when a change in the zoom ratio is detected, the cropping step is moved forward between obtaining the original image and image signal processing (ISP processing), and the original image is cropped according to the zoom ratio before image signal processing. Since the cropped image has fewer pixels than the original image, fewer pixels need to be processed by ISP, which can reduce the ISP processing time, increase the ISP processing speed, and thus increase the shooting speed and user experience. In addition, during the ISP processing process, the pixels of the image need to occupy memory. Since the cropped image has fewer pixels than the original image, the memory required during the processing is reduced, which can optimize the memory and thus improve the overall shooting performance.

[0007] In combination with the first aspect, in some implementations of the first aspect, when the electronic device crops the first image according to the second zoom ratio, it can determine a first center point of the first image; and crops the first image according to the first center point and the second zoom ratio. This implementation performs cropping based on the center point, so that the enlarged image matches the image previewed by the user.

[0008] In combination with the above-mentioned first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, the electronic device also includes a second camera, the second camera corresponds to the first zoom ratio, and the first camera corresponds to the second zoom ratio. After responding to the first operation, the second camera is switched to the first camera. When the electronic device crops the original image according to the first center point and the second zoom ratio, it can determine the second center point corresponding to the preview image obtained by the second camera based on the first zoom ratio before the switching occurs; and crop the original image according to the first center point, the second zoom ratio and the second center point.

[0009] The electronic device includes a first camera and a second camera. When the zoom ratio is the first zoom ratio, the electronic device uses the second camera to obtain a preview image and displays it. When the first zoom ratio is adjusted to the second zoom ratio, the camera is switched from the second camera to the first camera, and the electronic device uses the first camera to obtain an original image. Since the positions of the optical centers of the first camera and the second camera are different, the positions of the center points of the preview image and the original image are different. The original image is cropped according to the first center point, the second center point, and the second zoom ratio. The obtained first image can match the preview image when the first image is displayed, which is more in line with user expectations and improves user experience.

[0010] In combination with the first aspect and the implementation methods, in some implementation methods of the first aspect, after acquiring the original image, the electronic device can determine the readout format corresponding to the original image, and different readout formats correspond to different pixel merging methods of the original image; the electronic device can crop the original image according to the cropping strategy corresponding to the readout format. Since the formats of the original image read out by the image sensor are different, the electronic device crops the original image according to the cropping strategy corresponding to the readout format during cropping, which can achieve better cropping effects, thereby achieving better display effects and shooting effects.

[0011] In combination with the above-mentioned first aspect and the above-mentioned implementation method, in some implementation methods of the first aspect, in the process of performing image signal processing on the first image, the electronic device can determine the second matrix based on the second zoom ratio and the first matrix, the first matrix is ​​used to adjust the brightness of the original image, the second matrix is ​​used to adjust the brightness of the first image, and the second matrix is ​​a submatrix of the first matrix; according to the second matrix, image signal processing is performed on the first image.

[0012] ISP processing includes lens shading compensation. In this implementation, lens shading compensation can be performed on the first image after cropping, wherein the gain value for adjusting the image brightness can be recorded in matrix form, and the gain value is an element in the matrix. The electronic device pre-stores a first matrix corresponding to the original image. After the original image is cropped, the second matrix is ​​determined according to the second zoom ratio and the first matrix, so that the second matrix is ​​adapted to the first image. The brightness value of the first image is adjusted using the second matrix, so that the brightness of the first image can be more uniform and the shooting effect can be better.

[0013] In combination with the first aspect and the implementation manner, in some implementation manners of the first aspect, performing image signal processing on the first image according to the second matrix includes: processing the second matrix according to the first algorithm; and performing image signal processing on the first image according to the processed second matrix. In this implementation manner, the number of elements in the processed second matrix is ​​increased compared to the second matrix before processing, which can make the brightness of the first image more uniform and achieve better shooting effect.

[0014] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; one or more memories; the memory stores one or more programs, and when one or more programs are executed by the processor, the electronic device executes any possible method in the above-mentioned first aspect.

[0015] In a third aspect, an embodiment of the present application provides a device, which is included in an electronic device, and the device has the function of realizing the above-mentioned aspects and the behavior of the electronic device in the possible implementation methods of the above-mentioned aspects. The function can be implemented by hardware, or the corresponding software can be implemented by hardware. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the method described in the first aspect above.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect above.

[0018] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of an image processing flow provided in an embodiment of the present application is shown;

[0021] Figure 3 Another example of an image processing flow diagram provided in an embodiment of the present application is shown;

[0022] Figure 4 Another example of an image processing flow diagram provided in an embodiment of the present application is shown;

[0023] Figure 5 A cutting schematic diagram provided in an embodiment of the present application is shown;

[0024] Figure 6 A schematic diagram of a camera of an electronic device provided in an embodiment of the present application is shown;

[0025] Figure 7 Another cutting schematic diagram provided in an embodiment of the present application is shown;

[0026] Figure 8 Another schematic diagram of a scenario provided by an embodiment of the present application is shown;

[0027] Fig. 9 A schematic diagram showing a grid method for lens shading correction provided in an embodiment of the present application

[0028] Fig.10 Another example of an image processing flow diagram provided in an embodiment of the present application is shown;

[0029] Fig.11 A schematic diagram showing another example of a grid method for lens shading correction provided in an embodiment of the present application is shown.

[0030] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0031] Fig.13 A software structure block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings. Hereinafter, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0033] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0034] 1. Focal length. The focal length indicates the size of the refractive power. The shorter the focal length, the greater the refractive power. The focal length of the optical lens assembly determines the size of the image generated by the object photographed by the optical lens assembly on the imaging plane. Assuming that the same object is photographed at the same distance, the longer the focal length of the optical lens assembly, the greater the magnification of the image generated by the object on the photosensitive element (charge-coupled device, CCD).

[0035] 2. Optical zoom, mainly the comparison ratio and switching of different focal lengths in the camera module. The optical zoom ratio can be used to represent the ability of optical zoom. The larger the optical zoom ratio, the farther the scene can be photographed. The size of the optical zoom ratio is related to the physical focal length of the optical lens assembly. The equivalent focal length of the camera module is usually 28mm, corresponding to 1× (i.e. 1x) optical zoom ratio.

[0036] 3. RAW image: RAW image refers to the original image obtained by converting the captured light source signal into a digital signal by the image sensor in the electronic device. The RAW data collected by the sensor is usually 10 bits, and it takes two bytes to store RAW data. In order to save storage space, the data can be stored in MIPI RAW format. MIPI RAW uses 5 bytes, a total of 40 bits to store 4 RAW data. Manufacturers usually process the MIPIRAW data, including but not limited to bad pixel compensation, certain noise reduction, black field calibration, green balance, preliminary vignetting compensation, etc. The RAW image generated after these processes is called Ideal RAW.

[0037] 4. Image signal processing (ISP): mainly used to process the digital signal output by the front-end image sensor.

[0038] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

[0039] The image processing method provided in the embodiment of the present application can be applied to various electronic devices. The electronic device can be, but is not limited to, a mobile sports camera (GoPro), a digital camera, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, a mobile phone, etc., and the embodiment of the present application does not limit this.

[0040] The following description will take a mobile phone as an example. Currently, all mobile phones have a shooting function. When a user uses a mobile phone to shoot an image, he or she can select the focal length required for shooting to reduce or enlarge the image. The following description will be made on the shooting zoom scene. Figure 1 A schematic diagram of a shooting interface provided in an embodiment of the present application.

[0041] like Figure 1 As shown in (a) of FIG. 1 , a camera application is installed on the mobile phone. Exemplarily, in response to a user clicking on the camera application, the mobile phone runs the camera application and displays the following Figure 1 The shooting interface shown in (b).

[0042] The shooting interface includes multiple options for providing users with different shooting modes. For example, large aperture mode 11, night scene mode 12, portrait mode 13, photo mode 14, video mode 15, professional mode 16, etc. The shooting interface also includes a shooting key 20, which is used to indicate the current shooting mode. For example, Figure 1 As shown in (b) in FIG. 1 , the shooting key 20 indicates that the current shooting mode is the photo mode 14 .

[0043] The shooting interface also includes a viewfinder window 30, which can be used to display a preview image before taking a picture in real time. In addition, a zoom option 31 is also displayed in the shooting interface. The user can select the zoom ratio required for the current shooting in the zoom option 31. For example, 0.5 times, 2 times, 3.5 times or 50 times zoom ratio, etc. Among them, the zoom ratio can be understood as the focal length of the current camera, which is equivalent to the zoom multiple of the base focal length. The zoom ratio in each embodiment of the present application can also be referred to as the ratio.

[0044] The user may perform a first operation to select a zoom ratio to adjust the preview image displayed in the viewfinder window 30. The first operation may include a gesture operation for zooming the preview image, a screen click operation, or a sliding zoom progress bar. Figure 1 As shown in (c) of FIG. 1 , the user can increase the zoom ratio by sliding the zoom option 31 upwards. At this time, the photographed object is continuously enlarged on the viewfinder 30. After the user stops sliding, as shown in FIG. Figure 1 As shown in (d) in the figure, the enlarged preview image generated at the second zoom ratio is displayed, and the corresponding zoom ratio after the user stops sliding is the second zoom ratio.

[0045] In a typical zoom shooting scenario, the electronic device processes the original image as follows: Figure 2 As shown. The image sensor obtains the MIPI RAW image, and after some processing (such as decompression (unpack), black level correction) of the MIPI RAW image, the Ideal RAW image is obtained, and the Ideal RAW image is sent to the image signal processor (image signal processor, ISP) for related algorithm processing. After processing the entire photo, the image is cropped according to the zoom ratio, and the cropped image is displayed on the preview interface. The image format can be JPEG (Joint Photographic Experts Group). Among them, the image sensor is an important part of the camera.

[0046] This processing method causes a certain loss in memory and performance of electronic devices. No matter how many times the user zooms in to shoot, the image sensor will output the original image according to the initial zoom ratio (for example, 1×). The back-end ISP processes the original image (larger in size), and after being processed by the ISP, part of the pixels of the original image will be cropped. It is actually unnecessary for the ISP to process the cropped pixels, so processing unnecessary pixels will increase the processing time and cause corresponding power consumption loss. Moreover, these pixels that need to be cropped will occupy memory, which puts pressure on the memory of electronic devices.

[0047] In view of this, the present application embodiment provides an image processing method, such as Figure 3 As shown, when the user uses the shooting function of the electronic device, the user adjusts the first zoom ratio to the second zoom ratio. After the electronic device detects the user operation, in response to the user operation, the image sensor of the electronic device obtains the original image based on the first zoom ratio. Thereafter, the electronic device first crops the original image according to the second zoom ratio, and then performs ISP processing on the cropped image to obtain the zoomed image, which is displayed on the shooting interface.

[0048] In an embodiment of the present application, in a zoom shooting scene, when a change in the zoom ratio is detected, the cropping step is moved forward between obtaining the original image and image signal processing (ISP processing), and the original image is cropped according to the zoom ratio before image signal processing. Since the cropped image has fewer pixels than the original image, fewer pixels need to be processed by ISP, which can reduce the ISP processing time, increase the ISP processing speed, and thus increase the shooting speed and user experience. In addition, during the ISP processing process, the pixels of the image need to occupy memory. Since the cropped image has fewer pixels than the original image, the memory required during the processing is reduced, which can optimize the memory and thus improve the overall shooting performance.

[0049] In combination with the above content, the present application embodiment provides an image processing method, such as Figure 4 As shown, the shooting method may include the following S401 to S404.

[0050] S401: A first operation is detected, wherein the first operation requests adjusting a first zoom ratio to a second zoom ratio.

[0051] Exemplary, combined Figure 1 As shown in (a) and (b) of FIG. 1 , after the camera application is started, the camera displays a preview image based on a first zoom ratio. The first zoom ratio is the zoom ratio of the electronic device currently displaying the preview image. For example, Figure 1 The shooting interface shown in (b) is the initial interface when the user opens the camera application, and the first zoom ratio can be the initial zoom ratio "1×" (1 times). The second zoom ratio is the magnification ratio desired by the user, that is, the zoom ratio when the first operation finally stops. Figure 1 As shown in (c) and (d) in the figure, the first operation of the user on the zoom option can be a sliding operation on the zoom option, and the electronic device can determine the zoom ratio selected by the user according to the sliding direction (sliding up and down) and amplitude. For example, if the user slides the zoom option upward to 5×, the second zoom ratio is 5×. It should be noted that in the embodiment of the present application, the second zoom ratio is greater than the first zoom ratio.

[0052] S402, in response to a first operation, acquiring an original image, where the original image is acquired by a camera of the electronic device based on a first zoom ratio.

[0053] The camera acquires an original image based on the first zoom ratio, and the original image may be a MIPI RAW image or an Ideal RAW image.

[0054] S403: crop the original image according to the second zoom ratio to obtain a first image.

[0055] In response to the first operation, the electronic device determines a second zoom ratio, and then crops the original image according to the second zoom ratio to obtain a first image, wherein the size of the first image is smaller than that of the original image.

[0056] S404: Perform image signal processing on the first image.

[0057] The first image is still a MIPIRAW or Ideal RAW image. After the first image is obtained, some ISP processing is performed on the first image. In the process of ISP processing, each module is often used to process the image. The image can be processed in turn by black level compensation (BLC), lens shading correction (Lsc), bad pixel correction, demosaic, automatic white balance (AWB), color correction, gamma correction, etc.

[0058] It should be understood that Figure 3 The processing flow when the original image is an Ideal RAW image is exemplarily described. When the original image is a MIPI RAW image, the MIPI RAW image can be cropped first, and then the cropped image can be subjected to black level correction and subsequent processing. It should be understood that the timing change of the cropping step in the embodiment of the present application will not lose image quality.

[0059] S405, displaying an image after image signal processing is performed on the first image.

[0060] Display the image processed by ISP as Figure 1 The user can click the shooting key 20, and in response to the user's click operation, the electronic device can save the preview image shown in (d). Figure 1 The preview image shown in (d) in FIG.

[0061] In one implementation, the original image is in a default format, and the electronic device crops the original image according to a cropping strategy corresponding to the default format to obtain a first image.

[0062] In another implementation, after acquiring the original image, the electronic device determines a readout format of the original image; and crops the original image according to a cropping strategy corresponding to the readout format to obtain a first image.

[0063] In the embodiment of the present application, the electronic device 100 can adjust the pixel signal merging method of the image sensor according to the zoom ratio, so that the image sensor can read out the original image in different formats, such as reading out the first image in Binning format or Quadra format.

[0064] For example, when the zoom ratio range is [1, 5), the original image can be read out in Binning format, where Binning format refers to synthesizing the illumination corresponding to several adjacent pixels of the same color into one pixel for output. When the zoom ratio range is [5, 100), the original image can be read out in Quadra format, where Quadra format refers to arranging four pixels of the same color together in a 2×2 matrix, and when every four pixels of the same color are viewed as a whole, the Bayer format structure is satisfied.

[0065] Different image formats correspond to different cropping strategies. The cropping strategies corresponding to the Binning format are shown in equations (1) to (4).

[0066]

[0067] w=W-2x (3)

[0068] h=H-2y (4)

[0069] The cropping strategies corresponding to the Quadra format are shown in equations (5) to (8).

[0070]

[0071] w=W-2x (7)

[0072] h=H-2y (8)

[0073] Among them, zoomRatio is the zoom ratio, such as Figure 5 As shown, the larger rectangular box is the range where the original image is located, the smaller rectangular box is the range where the first image is located, W is the width of the original image (Width), H is the height of the original image (High), w is the width of the first image, h is the height of the first image, x is the margin between the original image and the first image in the width direction, and y is the margin between the original image and the first image in the height direction. The center points of the original image and the first image are both O.

[0074] The Binning format is used as an example for explanation below. In one implementation, the electronic device determines the first center point of the original image, and crops the original image according to the first center point and the second zoom factor. Specifically, the center point of the original image is used as the center point of the first image, the size of the first image is determined according to the cropping strategy corresponding to the Binning format, and the original image is cropped to obtain the first image.

[0075] For example, when the second zoom ratio is 2×, the width (W) and height (H) of the original image are 500*300 pixels, then the width and height of the first image are 1 / 2 of the original image, and w*h is 250*150 pixels. By comparison, the pixel area of ​​the first image is 1 / 4 of the original image, and 3 / 4 of the pixel area of ​​the original image can be optimized in memory.

[0076] In another implementation, considering that the electronic device may include multiple cameras, multiple cameras may be switched during the zoom process, and the center points corresponding to different cameras are different, the electronic device determines the second center point corresponding to the camera before switching and the first center point corresponding to the camera after switching, and crops the original image according to the first center point, the second center point and the second zoom ratio. The obtained first image can match the preview image when the first image is displayed, better meet user expectations, and improve user experience.

[0077] The multiple cameras may include an ultra-wide-angle camera, a main camera, and a telephoto camera. Optionally, the multiple cameras may also include at least one camera such as a depth camera, a black and white camera, and a multi-spectral camera.

[0078] For example, Figure 6 As shown, the mobile phone is provided with three cameras, which may be a main camera 61, an ultra-wide-angle camera 62, and a telephoto camera 63, respectively, and may be arranged in the upper left corner of the back cover of the mobile phone. Among them, the main camera 61, as a camera commonly used by users, is separately distributed in the first row near the top of the back cover, and the ultra-wide-angle camera 62 and the telephoto camera 63 are located below the main camera 61. Of course, these three cameras can also be composed of other cameras. The above is only an example, and the embodiment of the present application does not impose any restrictions on this. When the mobile phone is shooting, these three cameras are used to shoot the same scene to be shot.

[0079] It should be understood that the main camera 61 is generally used as the default camera of the electronic device. When the electronic device responds to the user's operation to start the camera, the main camera 61 can be started by default, and then the image captured by the main camera 61 is displayed in the preview interface. It should be noted that the main camera 61 can also be called a wide-angle camera. The ultra-wide-angle camera 62 has a larger field of view and is suitable for shooting larger subjects, such as scenery. The telephoto camera 63 has a longer focal length and a smaller field of view, and is suitable for shooting subjects that are far away from the mobile phone, that is, distant objects.

[0080] Among them, the zoom ratio corresponding to the ultra-wide-angle camera 62 is relatively smaller than the zoom ratio of the main camera 61, and the zoom ratio of the telephoto camera 63 is relatively larger than the zoom ratio of the main camera 61. Exemplarily, the zoom ratio range corresponding to the ultra-wide-angle camera 62 is [0.1, 1), where 0.1 refers to a 0.1x zoom ratio and 1 refers to a 1x zoom ratio. The zoom ratio range corresponding to the main camera 61 is [1, 5), and the zoom ratio range corresponding to the telephoto camera 63 is [5, 100).

[0081] It should be understood that there is a corresponding relationship between the zoom ratio range and the camera, and different zoom ratio ranges correspond to different cameras. The electronic device selects a suitable camera from multiple cameras as the target camera for shooting according to the range to which the second zoom ratio belongs.

[0082] For example, the user uses the first zoom ratio of 1× in the photo mode and uses the main camera 61 to take pictures. If the second zoom ratio is 2× after focusing, and the second zoom ratio belongs to the zoom ratio range applicable to the main camera 61, the main camera 61 is used as the target camera to obtain the original image. In this case, the first camera and the second camera are both the main camera 61, the first center point and the second center point are the same point, corresponding to the optical center position of the main camera 61, and the cropping method is referred to Figure 5 The embodiment shown.

[0083] If after the user adjusts the focus, the second zoom ratio is 20×, which belongs to the zoom ratio range [5, 100) applicable to the telephoto camera 63, the telephoto camera 63 is used as the target camera to obtain the original image. In this case, the first camera is the telephoto camera 63, the second camera is the main camera 61, the first center point corresponds to the optical center position of the telephoto camera 63, and the second center point corresponds to the optical center position of the main camera 61. The electronic device can determine the optical center offset according to the first center point and the second center point, determine the first area according to the second zoom ratio, and offset the first area according to the optical center offset to obtain the second area, and the second area corresponds to the first image. The optical center offset indicates the relative displacement of the optical center positions of the two cameras. The optical center offset is a known calibration quantity.

[0084] For example, Figure 7 As shown, before the camera is switched, the main camera 61 obtains a preview image based on the first zoom ratio (1×), and the preview image is a rectangular area surrounded by vertices P1, P2, P3, and P4, and the second center point is O2. After the camera is switched, the telephoto camera 63 obtains the original image based on the first zoom ratio (1×), and the original image is a rectangular area surrounded by vertices M1, M2, M3, and M4, and the first center point is O1. The preview image after the zoom ratio is changed is a rectangular area (second area) surrounded by r1, r2, r3, and r4, and the center point is O2. After obtaining the original image, the rectangular area (first area) surrounded by vertices m1, m2, m3, and m4 is determined based on the second zoom ratio and the corresponding cropping strategy with O1 as the center. In order to make the cropped image after the zoom ratio changes consistent with the user's preview image, the first area needs to be close to the second area. After calculating the positions of the four points m1, m2, m3, and m4 in the first image, the positions of the four vertices r1, r2, r3, and r4 in the original image are calculated according to the optical center offset O1O2 and the positions of the four vertices of the first area, so as to determine the second area. The image included in the second area is the required first image, and the rest of the original image except the second area is all cropped.

[0085] In the embodiment of the present application, the electronic device can crop the original image before any algorithm in the ISP processing flow, for example, it can crop before black level compensation, it can also crop after black level compensation and before lens shading correction, it can also crop after lens shading correction and before bad pixel correction, etc. For example, Figure 8 As shown, the electronic device crops the original image between black level compensation and lens shading correction to obtain a cropped image (first image), and then performs lens shading correction and subsequent ISP processing on the first image.

[0086] Among them, lens occlusion is a common optical phenomenon. When light passes through the edge of the lens, it will be blocked or scattered, causing the corners or edges of the image to become dark. Fig. 9 As shown in the figure, lens shading is divided into brightness shading (Luma shading) and color deviation (Color shading). The reason for brightness shading is that the amount of light transmitted by the lens decreases from the center to the corners, resulting in the image looking normal in the middle and dark around. Color deviation is called color non-uniformity. The transmission capacity of the three primary colors of RGB is different at different color temperatures, which manifests as uneven image color when taking pictures. The purpose of lens shading correction is to make the brightness of the entire image more uniform.

[0087] Lens shading correction can be performed by using storage gain method, polynomial fitting method, automatic correction method, etc. Taking storage gain method as an example, the storage gain method multiplies the brightness value of each pixel on the image by a certain value, where pixels at different positions correspond to different values, thereby increasing the brightness value of pixels in dark places. These values ​​can be called gain values.

[0088] The gain value is pre-calibrated and stored in the electronic device, and can be called in real time when shooting. Since there are many pixels in an image, storing the gain value for each pixel takes up too much memory. In the embodiment of the present application, the entire image is divided into several grids, and the gain value is stored for each grid. For example, Fig.10 As shown in (a) in , the image can be divided into grids with sparse grids in the middle and dense grids around. Each grid corresponds to a gain value, and the pixels in each grid use the same gain value. Fig.10 As shown in (b), the image can be divided into n*n grids, and the gain value of each grid vertex is pre-calibrated, and the gain value of the pixel point located in the grid is determined by interpolation calculation. The electronic device can store the gain value in the form of a table, a matrix, etc. For example, the gain value is stored in the form of a matrix, and the gain value corresponding to each grid or vertex is an element in the matrix, and the gain value of the grid or vertex in the nth row and nth column is the element in the nth row and nth column of the matrix.

[0089] In one implementation, when performing lens shading correction on the first image, a pre-stored gain value may be used, according to the original grid (eg Fig. 9 The first image is divided into 13×17 grids as shown in (a) in FIG. 1 , and the pixels in the grid are corrected using the gain value corresponding to each grid. The original grid corresponds to the first matrix.

[0090] In another implementation, the original grid (lsctable) may be divided according to the image size corresponding to the initial zoom factor. When performing lens shading correction on the first image, the original grid is cropped according to the cropping method of the first image to obtain a cropped grid corresponding to the size of the first image. For example, the cropped grid is Fig. 9 As shown by the dotted line in (a) in FIG. 1 . The cropped grid is divided into 6×8 grids, and the first image is divided into 6×8 grids according to the cropped grid, and the pixels in the grid are corrected using the gain value corresponding to each grid. The cropped grid corresponds to the second matrix. It should be understood that the cropped grid is a part of the original grid, and the second matrix is ​​a submatrix of the first matrix. The elements in the mth row and nth column of the second matrix are the same as the elements in the sth row and pth column of the first matrix, s≥m, p≥n. This implementation method can make the image brightness more uniform and the processing effect better.

[0091] In another implementation, when performing lens shading correction on the first image, the original table is cropped according to the cropping method of the first image to obtain a cropped grid corresponding to the size of the first image, the cropped grid is processed by the first algorithm, and the first image is corrected according to the processed grid. The first algorithm is not limited, for example, it can be a bilinear interpolation method, and the cropped grid can be made denser by bilinear difference to obtain an interpolated grid. The gain corresponding to the interpolated grid is used to act on the first image through the B channel, gr channel, g channel, and br channel to correct the pixels of the first image. The interpolated grid corresponds to the processed second matrix, and accordingly, the number of elements in the processed second matrix increases. This implementation can make the image brightness more uniform, the transition smoother, and the processing effect better.

[0092] The principle of bilinear interpolation is as follows Fig.11 As shown, insert a point P = (x, y) and find the value of point P. 11 =(x1,y1),Q 12 =(x1,y2),Q 21 =(x2,y1) and Q 22 =(x2,y2) The coordinates and values ​​of the four points are first interpolated linearly in the x direction to obtain the values ​​of R1 and R2, and then linearly interpolated in the y direction to obtain the value of P. The result of linear interpolation is independent of the order of interpolation. It is also possible to interpolate in the y direction first and then in the x direction. The specific calculation process is shown in equations (9) to (11). Among them, Q 11 , Q 12 , Q 21 , Q 22 It can correspond to four grids respectively, for example, the center point of the grid, and the inserted point P is a newly added grid. For example, point P is the center point of the newly added grid, and the value of point P is to determine the gain value of the newly added grid based on the gain values ​​of the known four grids.

[0093]

[0094] In summary, in an embodiment of the present application, in a zoom shooting scene, when a change in the zoom ratio is detected, the cropping step is moved forward between the acquisition of the original image and the image signal processing (ISP processing), and the original image is cropped according to the zoom ratio before image signal processing. Since the cropped image has fewer pixels than the original image, fewer pixels need to be processed by ISP, which can reduce the ISP processing time, increase the ISP processing speed, and thus increase the shooting speed and user experience. In addition, during the ISP processing process, the pixels of the image need to occupy memory. Since the cropped image has fewer pixels than the original image, the memory required during the processing is reduced, which can optimize the memory and thus improve the overall shooting performance.

[0095] The following is a schematic diagram of the hardware structure of an electronic device 100 that can implement the method provided by the present application. Fig.12 A hardware structure diagram of an electronic device 100 provided in an embodiment of the present application is shown.

[0096] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0097] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0098] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0099] Among them, the controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and execution.

[0100] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0101] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0102] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0103] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0104] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0105] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0106] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0107] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0108] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0109] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0110] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0111] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0112] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0113] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0114] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0115] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0116] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0117] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0118] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0119] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0120] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0121] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0122] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0123] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0124] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0125] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0126] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0127] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0128] The electronic device 100 can implement audio functions such as music playing, recording, voice calls, video calls, etc. through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0129] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0130] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0131] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0132] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0133] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0134] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0135] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0136] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0137] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0138] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0139] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0140] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0141] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0142] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0143] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0144] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0145] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0146] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0147] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0148] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0149] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0150] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 100. It should be noted that in the embodiment of the present application, the operator system of the electronic device may include but is not limited to (Symbian), (Android), (iOS), (Blackberry), Hongmeng (HarmonyOS) and other operating systems, this application does not make any limitation. Fig.13 A software structure block diagram of the electronic device 100 provided in an embodiment of the present application is shown.

[0151] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into the application layer, application framework layer (Application Framework), hardware abstract layer (HAL), kernel layer, and hardware layer from top to bottom.

[0152] The application layer may include a camera application or other applications, including but not limited to: music, gallery, Bluetooth, calendar, short message, call, navigation and other applications.

[0153] The application framework layer may provide an application programming interface (API) and a programming framework to the application programs of the application layer; the application framework layer may include some predefined functions.

[0154] For example, the application framework layer may include a camera access interface; the camera access interface may include camera management and camera equipment; wherein, camera management may be used to provide an access interface for managing the camera; and camera equipment may be used to provide an interface for accessing the camera.

[0155] The hardware abstraction layer is used to abstract the hardware. For example, the hardware abstraction layer can include the camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer includes the ISP algorithm, etc. After the camera's image sensor obtains the RAW data, the ISP algorithm is used to crop, enhance, optimize, and other processes on the RAW data to obtain the zoomed image. In addition, the camera abstraction layer also stores the captured image in the gallery application for users to view. The camera hardware abstraction layer is also used to determine the target camera and the output format according to the zoom ratio selected by the user, and issue corresponding instructions.

[0156] The kernel layer is used to provide drivers for different hardware devices. For example, the driver layer may include camera drivers, display drivers, and image processor drivers.

[0157] The hardware layer may include multiple image sensors, multiple image signal processors, cameras, display screens, and other hardware devices.

[0158] In this application, by calling the hardware abstraction layer interface in the hardware abstraction layer, the application layer and application framework layer above the hardware abstraction layer can be connected with the driver layer and hardware layer below, thereby realizing camera data transmission and function control.

[0159] In the camera hardware interface layer in the hardware abstraction layer, manufacturers can customize functions according to their needs. Compared with the hardware abstraction layer interface, the camera hardware interface layer is more efficient, flexible, and has low latency. It can also call ISP and GPU more richly to achieve image processing. The image input to the hardware abstraction layer can come from an image sensor or a stored picture.

[0160] The scheduling layer in the hardware abstraction layer contains a general functional interface for management and control.

[0161] The camera service layer in the hardware abstraction layer is used to access the interface of the ISP and other hardware.

[0162] The following is an example of the workflow of the software and hardware of the electronic device 100 in conjunction with the photo zoom scenario.

[0163] The camera application in the application layer can be displayed on the screen of the electronic device 100 in the form of an icon. When the icon of the camera application is clicked by the user to trigger it, the electronic device 100 starts to run the camera application. When the camera application runs on the electronic device 100, the camera application calls the interface corresponding to the camera application in the application framework layer, and then starts the camera driver by calling the hardware abstraction layer, turns on the multiple cameras 193 on the electronic device 100, and obtains the original image through the target camera. If the camera application detects the user's focusing operation, it calls the interface corresponding to the camera application in the application framework layer, calls the target camera corresponding to the second zoom ratio, and obtains the original image based on the first zoom ratio; then, by calling the ISP algorithm, the original image is cropped according to the second zoom ratio to obtain the first image, and the first image is subjected to ISP processing such as lens shading compensation, and then the cropped and enlarged image is output to the display screen for display.

[0164] The electronic device provided in this embodiment is used to execute the above method, so the same effect as the above implementation method can be achieved. In the case of an integrated unit, the electronic device may include a processing module, a storage module and a communication module. Among them, the processing module can be used to control and manage the actions of the electronic device, for example, it can be used to support the electronic device to execute the steps performed by the processing unit. The storage module can be used to support the electronic device to execute stored program codes and data, etc. The communication module can be used to support the communication between the electronic device and other devices.

[0165] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0166] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0167] In addition, the embodiments of the present application also provide a device, which can be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory, so that the chip executes the methods in the above-mentioned method embodiments. The chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGA), programmable logic devices (PLD), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout the present application.

[0168] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In the above description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0173] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0174] It should also be understood that the "multiple" mentioned in the specification and the appended claims of this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, which means any combination of one or more of the items listed in the association and all possible combinations, and includes these combinations, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0175] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0176] In addition, in order to clearly describe the technical solution of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and cannot be understood as indicating or implying relative importance, and words such as "first" and "second" do not necessarily limit differences.

[0177] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0178] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: A first operation is detected, wherein the first operation requests to adjust the first zoom ratio to a second zoom ratio; In response to the first operation, an original image is acquired, where the original image is acquired by a first camera of the electronic device based on the first zoom ratio; Cropping the original image according to the second zoom ratio to obtain a first image; performing image signal processing on the first image; displaying an image after image signal processing is performed on the first image; The performing image signal processing on the first image includes: determining a second matrix according to the second zoom ratio and the first matrix, the first matrix being used to adjust the brightness of the original image, the second matrix being used to adjust the brightness of the first image, and the second matrix being a submatrix of the first matrix; and performing image signal processing on the first image according to the second matrix; The first matrix includes an original grid, and the second matrix includes a cropped grid; determining the second matrix according to the second zoom ratio and the first matrix includes: cropping the original grid according to a cropping method of the first image to determine the cropped grid, wherein the cropping method of the first image is determined according to the second zoom ratio; The performing image signal processing on the first image according to the second matrix includes: Processing the second matrix according to the first algorithm, so that the number of elements in the second matrix after the processing increases compared with that in the second matrix before the processing; Performing image signal processing on the first image according to the processed second matrix.

2. The method according to claim 1, characterized in that: The cropping of the original image according to the second zoom ratio includes: Determining a first center point of the original image; The original image is cropped according to the first center point and the second zoom ratio.

3. The method according to claim 2, characterized in that: The electronic device further includes a second camera, the second camera corresponds to the first zoom ratio, the first camera corresponds to the second zoom ratio, and in response to the first operation, the second camera is switched to the first camera, and the cropping of the original image according to the first center point and the second zoom ratio includes: Determine a second center point corresponding to the preview image acquired by the second camera based on the first zoom ratio before the switching occurs; The original image is cropped according to the first center point, the second zoom ratio, and the second center point.

4. The method according to any one of claims 1 to 3, characterized in that After acquiring the original image, the method further includes: Determine a readout format corresponding to the original image, where different readout formats correspond to different pixel merging modes of the original image; The cropping of the original image comprises: The original image is cropped according to a cropping strategy corresponding to the readout format.

5. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 4.

Images