A photographing method and an electronic device

Abstract

The embodiment of the application relates to the terminal technical field, and in particular to a shooting method and electronic equipment, which can continuously display preview images for a user, and help to improve the user experience. The method comprises the following steps: displaying a preview interface of a camera application; displaying a first preview image in the preview interface according to a first RAW graph output by an image sensor; a pixel array of the first RAW graph is a standard Bayer array; a resolution of the first RAW graph is 1 / N of a resolution of the image sensor; in response to a received shooting operation, displaying a second preview image in the preview interface according to a second RAW graph output by the image sensor, and saving a shooting image; a pixel array of the second RAW graph is an N-Bayer pixel array; and a resolution of the second preview image is less than the resolution of the image sensor.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a shooting method and an electronic device. Background Technology

[0002] With the continuous development of terminal technology, cameras in electronic devices are increasingly using N-in-one image sensors. Cameras using N-in-one image sensors can capture high-resolution images in bright shooting environments and high-brightness images in dim shooting environments.

[0003] The pixels of an N-in-1 image sensor are arranged in an N-Bayer pixel array, such as a four-Bayer pixel array, a nine-Bayer pixel array, or a sixteen-Bayer pixel array. When using an electronic device with a camera that includes an N-in-1 image sensor, in response to a photo-taking operation, the preview image in the camera application's preview interface stops displaying, and the screen goes black, leading the user to mistakenly believe that the electronic device is lagging. Summary of the Invention

[0004] This application provides a shooting method and an electronic device that can continuously display preview images to users, thereby improving the user experience.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, a shooting method is provided, applied to an electronic device, the electronic device including an image sensor, the pixel array of the image sensor being an N Bayer pixel array, where N is an integer greater than 1. The method includes: displaying a preview interface of a camera application; displaying a first preview image on the preview interface based on a first RAW image output by the image sensor; the pixel array of the first RAW image being a standard Bayer array; the resolution of the first RAW image being 1 / N of the resolution of the image sensor; and, in response to a received shooting operation, displaying a second preview image on the preview interface based on a second RAW image output by the image sensor, and saving the captured image; the pixel array of the second RAW image being an N Bayer pixel array.

[0007] In this application, in a preview scene, the electronic device can display a first preview image on the preview interface based on the first RAW output from the image sensor. In response to a photo-taking operation, the electronic device enters a shooting scene, where it can display a second preview image on the preview interface based on the second RAW output from the image sensor. Specifically, the electronic device can convert the pixel arrangement of the second RAW output from the image sensor from an N-bay pixel array to a standard Bayer array. Then, the electronic device generates and displays a second preview image based on the converted image. In this way, even in a shooting scene, the electronic device can still display a preview image to the user on the camera application's preview interface, avoiding a black screen and improving the user experience.

[0008] In one possible implementation of the first aspect, the first RAW image is output by the image sensor in a first mode. The first mode is a binning operating mode. The pixel array of the first RAW image is a standard Bayer array, and the resolution of the first RAW image is lower than the resolution of the image sensor. Due to the lower resolution of the first RAW image, in a preview scene, the electronic device can display a preview image based on the first RAW image, meeting the requirement for real-time display of the preview stream.

[0009] In one possible implementation of the first aspect, the second RAW image is output by the image sensor in a second mode. In response to a received image capture operation, the electronic device switches the image sensor's operating mode from the first mode to the second mode, provided preset conditions are met. In the first mode, the image sensor outputs a RAW image with a standard Bayer pixel array and a resolution of 1 / N of the image sensor's resolution; in the second mode, the image sensor outputs a RAW image with an N Bayer pixel array; wherein meeting the preset conditions indicates that the electronic device needs to output a high-resolution captured image. In scenarios where the electronic device needs to output a high-resolution captured image, after entering the shooting scene, the electronic device can obtain and display a second preview image based on the second RAW image output by the image sensor, and obtain and save the captured image based on the second RAW image output by the image sensor. In this way, the electronic device can not only continue to display the preview image but also output a high-resolution captured image.

[0010] In one possible implementation of the first aspect, the preset condition is any one of the following: the current zoom level is greater than 1, the current ambient brightness is greater than a threshold, HDR function is enabled, high-resolution image output function is enabled, and the current shooting mode is professional mode.

[0011] In one possible implementation of the first aspect, the electronic device uses the pixel value of any one of N adjacent pixels of the same color in the second RAW image as the pixel value of a pixel in the third RAW image. Then, the electronic device obtains a second preview image based on the third RAW image and displays the second preview image on a preview interface. This implementation provides a feasible way for an electronic device to obtain a second preview image based on a second RAW image output by an image sensor.

[0012] In one possible implementation of the first aspect, the electronic device performs a target operation on the pixel values ​​of N adjacent pixels of the same color in the second RAW image, and uses the pixel value after the target operation as the pixel value of a pixel in the third RAW image; the target operation is any one of summation, weighted summation, or averaging. Then, the electronic device obtains a second preview image based on the third RAW image and displays the second preview image on a preview interface. This implementation provides a feasible way for an electronic device to obtain a second preview image based on a second RAW image output by an image sensor.

[0013] In one possible implementation of the first aspect, the electronic device inputs a second RAW image into a neural network model to obtain a third RAW image. Then, the electronic device obtains a second preview image based on the third RAW image and displays the second preview image on a preview interface. The neural network model has the ability to process a RAW image with an N-bay pixel array into a RAW image with a standard Bayer pixel array. This implementation provides a feasible way for an electronic device to obtain a second preview image based on a second RAW image output from an image sensor.

[0014] In one possible implementation of the first aspect, the neural network model is a first neural network model. The first neural network model has the ability to process a RAW image with an N-bay pixel array into a RAW image with a standard Bayer pixel array. Furthermore, the resolution of the processed RAW image is 1 / N of the resolution of the original RAW image.

[0015] In one possible implementation of the first aspect, the neural network model is a second neural network model. The second neural network model has the ability to process a RAW image with an N-bay pixel array into a RAW image with a standard Bayer pixel array. Furthermore, the resolution of the processed RAW image is the same as the resolution of the original RAW image.

[0016] In one possible implementation of the first aspect, the captured image is obtained from M second RAW images, where M is an integer greater than or equal to 1. After obtaining the M second RAW images, the image sensor's operating mode is switched from the second mode to the first mode, so that the RAW images output by the image sensor in the first mode can continue to be displayed on the preview interface. Thus, when switching from the shooting scene to the preview scene, the electronic device can continue to display the preview image based on the first RAW images, meeting the requirement for real-time display of the preview stream.

[0017] In a second aspect, an electronic device is provided, comprising: a memory, a camera, and one or more processors; the camera, the memory, and the processors are coupled; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any of the first aspects.

[0018] Thirdly, a chip system is provided that can be applied to an electronic device including memory. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The interface circuits are used to receive signals from the aforementioned memory and send the signals to the processors, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device performs a method as described in the first aspect and any of its possible design embodiments.

[0019] Fourthly, a computer-readable storage medium is provided, including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any of the first aspects.

[0020] Fifthly, a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of any of the methods in the first aspect.

[0021] In understanding, the beneficial effects that can be achieved by the electronic device of any possible design of the second aspect, the chip system of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect can be referred to as the beneficial effects of the first aspect and any possible design, which will not be repeated here. Attached Figure Description

[0022] Figure 1 Schematic diagrams of four common Bayer arrays provided for embodiments of this application;

[0023] Figure 2 A schematic diagram of a pixel array for a 4-in-1 image sensor provided in an embodiment of this application;

[0024] Figure 3 A schematic diagram of three formats of RAW images output by a 4-in-1 image sensor provided in an embodiment of this application;

[0025] Figure 4 A schematic diagram illustrating a shooting stream and a preview stream in an electronic device according to an embodiment of this application;

[0026] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0027] Figure 6 A schematic diagram illustrating the images output by an image sensor in two operating modes, as provided in an embodiment of this application;

[0028] Figure 7 A schematic flowchart illustrating a shooting method provided in an embodiment of this application;

[0029] Figure 8 A schematic diagram of a preview interface provided for an embodiment of this application;

[0030] Figure 9 A schematic diagram of a target processing method provided in an embodiment of this application;

[0031] Figure 10 This is a schematic diagram of the interaction between internal modules of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] Modern electronic devices generally have photo and video recording functions, allowing users to capture images or videos and record their lives anytime, anywhere.

[0033] Electronic devices, such as mobile phones, may include at least one camera. A camera may include a lens, holder, image sensor, voice coil motor (VCM), and flexible printed circuit (FPC), among other components. An image sensor, also known as a photosensitive device, is a device that converts optical images into electrical signals. For example, the surface of an image sensor contains hundreds of thousands to millions of photosensitive elements, also called pixels. When these photosensitive elements are illuminated, they generate light signals. The image sensor can convert these light signals into electrical signals and output an image.

[0034] A pixel is the smallest unit on an image sensor that senses light and converts it into an electrical signal. A pixel can capture the light signal of one color of light and convert it into an electrical signal. For example, pixel R captures the light signal of red light and converts it into an electrical signal; pixel R is a red pixel. Pixel G captures the light signal of green light and converts it into an electrical signal; pixel G is a green pixel. Pixel B captures the light signal of blue light and converts it into an electrical signal; pixel B is a blue pixel.

[0035] Pixels in an image sensor are typically arranged in a specific manner, such as a Bayer array. This Bayer array is also known as a standard Bayer array. Figure 1 The diagram illustrates four common Bayer arrays. These may include BGGR, GBRG, GRBG, and RGGB arrays. The following section uses the RGGB Bayer array as an example to introduce this solution.

[0036] With the continuous development of terminal technology, N-in-one image sensors have emerged. The pixel arrangement of an N-in-one image sensor can be an N-in-one Bayer array, also known as an N-pixel Bayer array. For example, an N-in-one Bayer array can include a 4-in-one Bayer array, a 9-in-one Bayer array, a 16-in-one Bayer array, etc. Taking a 4-in-one Bayer array as an example... Figure 2 As shown, pixel region A has 4 red pixels arranged in a 2×2 pattern, pixel region B has 4 green pixels arranged in a 2×2 pattern, pixel region C has 4 green pixels arranged in a 2×2 pattern, and pixel region D has 4 blue pixels arranged in a 2×2 pattern.

[0037] The N-in-1 image sensor can output three different formats of raw (RAW) images, referred to as the first format RAW image, the second format RAW image, and the third format RAW image, respectively. The pixel arrangement in the first format RAW image is consistent with the pixel arrangement of the N-in-1 image sensor. For example, the pixel arrangement in the first format RAW image is an N-in-1 Bayer array. The pixel arrangement in the second format RAW image is also a Bayer array. The resolution of both the first and second format RAW images is consistent with the resolution of the N-in-1 image sensor. The first and second format RAW images are also referred to as high-resolution RAW images. The third format RAW image uses a Bayer array for its pixels, and its resolution is lower than the resolution of the N-in-1 image sensor. For example, the resolution of the third format RAW image is 1 / N of the resolution of the N-in-1 image sensor.

[0038] Taking a 4-in-1 image sensor as an example, such as Figure 3As shown, the first format of the RAW image is as follows: Figure 3 As shown in (a) above. The first format RAW image output by the 4-in-1 image sensor can be called a quad RAW image, and the pixel arrangement in the quad RAW image is consistent with the pixel array of the 4-in-1 image sensor. The first format RAW image output by the 9-in-1 image sensor can be called a nona RAW image. The pixel arrangement in the nona RAW image is consistent with the pixel array of the 9-in-1 image sensor. The first format RAW image output by the 16-in-1 image sensor can be called a tetra 2 RAW image. tetra 2 The pixel arrangement in the RAW image is consistent with the pixel array of a 16-in-1 image sensor.

[0039] The second format of RAW chart is as follows: Figure 3 As shown in (b) of the diagram. The pixels in the second format RAW image are arranged in a Bayer array.

[0040] Electronic devices can use hardware remosaic processing to enable the N-in-1 image sensor to output a RAW image in a second format. Hardware remosaic processing modifies the pixel arrangement of the N-in-1 image sensor; for example, it can modify the pixel arrangement of the N-in-1 image sensor from an N-in-1 Bayer array to a Bayer array via hardware. After modification, the N-in-1 image sensor can output a RAW image in a second format.

[0041] Hardware remosaic processing requires specific hardware (such as specific logic circuits), resulting in high costs. Some electronic devices, in order to reduce costs, lack hardware remosaic processing capabilities. Optionally, in cases where some electronic devices do not support hardware remosaic processing, the N-in-1 image sensor cannot directly output a second-format RAW image. The electronic device can process the first-format RAW image into a second-format RAW image using software remosaic processing. Software remosaic processing can be performed by an ISP. For example, the ISP uses a remosaic algorithm to convert the first-format RAW image into a second-format RAW image. For example, the electronic device can use a software remosaic algorithm to... Figure 3 The first format RAW plot shown in (a) is converted to Figure 3 The second format RAW plot is shown in (b) above.

[0042] The third format RAW image is as follows Figure 3As shown in (c) above. In some embodiments, this RAW image is referred to as a binning image. When reading the electrical signals generated by each pixel, the N-to-one image sensor merges the electrical signals generated by adjacent n×n pixels of the same color into the electrical signal of the same pixel, thus obtaining a third-format RAW image.

[0043] After launching the camera app, it typically enters a preview mode. In this mode, the N-in-1 image sensor outputs a third-format RAW image for preview display by default. Generally, during a shooting scenario, the electronic device internally includes two data streams: a preview stream and a shooting stream. For example... Figure 4 As shown, in the capture stream, the image output by the image sensor is processed by the ISP to obtain the captured image, which the electronic device can save. In the preview stream, the image output by the image sensor is processed by the ISP to obtain the preview image, which the electronic device can display.

[0044] However, in some scenarios, when electronic devices lack hardware remosaic processing capabilities (or when hardware remosaic processing is not enabled), in response to a photo-taking operation, the electronic device stops displaying the preview image on the camera application's preview screen, and the display may go black. For example, in response to a photo-taking operation, the display may go black for a period of time, such as 0.5s-1s, which may lead users to believe that the preview interface is lagging.

[0045] Therefore, embodiments of this application provide a shooting method and an electronic device. In a shooting scenario, the electronic device can convert the pixel arrangement of the RAW image output by the image sensor, such as the second RAW, from an N-bay pixel array to a standard Bayer array. Then, the electronic device generates and displays a preview image based on the converted image. In this way, even during the shooting scenario, the electronic device can still display a preview image to the user in the camera application's preview interface, avoiding a black screen and improving the user experience.

[0046] The method provided in this application can be applied to electronic devices equipped with a display screen and a camera. These electronic devices may include mobile phones, tablets, laptops, netbooks, personal digital assistants (PDAs), in-vehicle devices, etc., and this application does not impose any limitations on them. In this application, the aforementioned electronic device is an electronic device capable of running an operating system and installing applications. Optionally, the operating system running on the electronic device may be... system, system, Systems, etc.

[0047] Take a mobile phone as an example. Figure 5As shown, the electronic device 500 may include: a processor 510, a memory 520, a universal serial bus (USB) interface 530, a power management module 540, antennas such as antenna 1 and antenna 2, a communication module 550, a display screen 560, an audio module 570, a camera 580, a sensor module 590, etc.

[0048] Processor 510 may include one or more processing units, such as application processors (APs), central processing units (CPUs), modem processors, graphics processing units (GPUs), ISPs, controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the electronic device 500. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0049] In this embodiment, the ISP is used to process the RAW image output by the image sensor. This processing converts the RAW image into a preview image or a captured image. The processing includes, but is not limited to, the first processing, second processing, and target processing described in this embodiment. The ISP may include an image front end (IFE) module, a first module, and an image processing engine (IPE) module connected in sequence. The IFE module performs first processing on the RAW image, including 3A processing. The first module performs target processing on the first-processed RAW image, modifying the pixel arrangement of the RAW image from an N-pixel Bayer array to a standard Bayer array. The IPE module performs second processing and other preset processing on the target-processed RAW image. The second processing may include demosaicing, and other processing may include white balance, color correction, etc. Demosaicing can convert the RAW image in the RAW domain into an image in the red, green, blue (RGB) domain.

[0050] The memory 520 can be used to store computer executable program code, which includes instructions. The processor 510 executes various functional applications and data processing of the electronic device by running the instructions stored in the memory 520. The memory 520 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as sound playback, interface display, etc.). The data storage area may store data created during the use of the electronic device (such as notification messages). Furthermore, the memory 520 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0051] In this embodiment, the memory 520 stores computer-executable program code, which includes instructions. The processor 510 executes a shooting method provided in this embodiment by running the instructions stored in the memory 520.

[0052] The power management module 540 is used to connect the battery to the processor 510. The power management module 540 receives battery and / or power input to power the processor 510, memory 520, communication module 550, display screen 560, and camera 580, etc. The power management module 540 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 540 may also be located within the processor 510.

[0053] The communication module 550 can provide solutions for wireless communication applications on the electronic device 500, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR). The communication module 550 can be one or more devices integrating at least one communication processing module. The communication module 550 receives electromagnetic waves via an antenna, frequency-modulates and filters the electromagnetic wave signal, and sends the processed signal to the processor 510. The communication module 550 can also receive signals to be transmitted from the processor 510, frequency-modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0054] In some embodiments, the antenna of the electronic device 500 is coupled to the communication module 550, enabling the electronic device 500 to communicate with networks and other devices via wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BitTorrent, Global Navigation Satellite System (GNSS), WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), BeiDou Navigation Satellite System (BDS), GLONASS, and / or Galileo.

[0055] Electronic device 500 implements display functions through a GPU, display screen 560, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 560 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 510 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0056] Display screen 560 is used to display images, videos, etc. Display screen 560 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0057] In this embodiment, the electronic device 500 can implement shooting and recording functions through an ISP, camera 580, video codec, GPU, display screen 560, and application processor.

[0058] The audio module 570 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. The audio module 570 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 570 may be located in the processor 510, or some functional modules of the audio module 570 may be located in the processor 510.

[0059] The camera 580 is used to capture still images or videos. An optical image of an object is projected onto a photosensitive element in an image sensor through the lens. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor converts the light signal into an electrical signal, which is then passed to an image ISP (Image Signal Processor) for conversion into a digital image signal. The image ISP outputs the digital image signal to a digital signal processing DSP (Digital Signal Processor). The DSP converts the digital image signal into standard image signals in formats such as RGB and YUV.

[0060] The sensor module 590 may include the aforementioned image sensor (such as an N-in-one image sensor), pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, proximity sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, and bone conduction sensor, etc.

[0061] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 500. In other embodiments, the electronic device 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] The following uses a mobile phone as an example to introduce a shooting method provided by an embodiment of this application.

[0063] A mobile phone includes a camera, an ISP, and a display. The camera includes an N-pixel image sensor. This N-pixel image sensor uses an N-pixel Bayer array. The pixels in the N-pixel image sensor can capture light signals and convert them into electrical signals. The N-pixel image sensor has two operating modes: a first mode and a second mode. In the first mode, the image sensor outputs a RAW image with a standard Bayer array pixel array and a resolution of 1 / N of the image sensor's resolution. In the second mode, the image sensor outputs a RAW image with an N-bayer pixel array pixel array. For example, in the first mode, the image sensor combines the electrical signals generated by adjacent n×n pixels of the same color into the electrical signal of one pixel, resulting in a first RAW image. In the second mode, the image sensor outputs the electrical signal generated by each pixel, resulting in a second RAW image.

[0064] In response to launching the camera app, the app enters the preview scene. The N-in-1 image sensor outputs a first RAW image in the first mode. The ISP processes the first RAW image into a first preview image, and the display screen can show a preview interface including this first preview image. In response to taking a photo, the camera app enters the shooting scene. The N-in-1 image sensor outputs a second RAW image in the second mode. The ISP processes the second RAW image into a second preview image, and the display screen can show a preview interface including this second preview image. The ISP can also obtain and save the captured image based on the second RAW image.

[0065] The first mode can be a Binning working mode. In the first mode, when the N-in-1 image sensor reads the electrical signals generated by each pixel, it merges the electrical signals generated by adjacent n×n pixels of the same color into the electrical signal of one pixel and reads it out, obtaining the first RAW image. The first RAW image can be a third-format RAW image, such as... Figure 3 As shown in (c) above. The second mode can be an N-Bayer operating mode. In the second mode, the N-in-1 image sensor reads out the electrical signal generated by each pixel individually to obtain a second RAW image. The second RAW image can be a RAW image in the first format, such as... Figure 3 As shown in (a) in the figure.

[0066] Taking a 4-in-1 image sensor as an example, such as Figure 6 As shown, in the first mode, the 4-in-1 image sensor adds together the electrical signals generated by the 2×2 red pixels included in pixel region A, and reads them out as a single red pixel signal. The 4-in-1 image sensor adds together the electrical signals generated by the 2×2 green pixels included in pixel region B, and reads them out as a single green pixel signal. The 4-in-1 image sensor adds together the electrical signals generated by the 2×2 green pixels included in pixel region C, and reads them out as a single green pixel signal. The 4-in-1 image sensor adds together the electrical signals generated by the 2×2 blue pixels included in pixel region D, and reads them out as a single blue pixel signal. This yields a first RAW image. The resolution of the first RAW image is lower than the resolution of the N-in-1 image sensor; specifically, the resolution of the first RAW image is 1 / N of the resolution of the N-in-1 image sensor. For example, if the resolution of the 4-in-1 image sensor is 8000×6000, the resolution of the first RAW image is 2000×1500. For example, a 9-in-1 image sensor has a resolution of 18000×10800, and its first RAW image output has a resolution of 2000×1200. Figure 6 As shown, in the second mode, the 4-in-1 image sensor sequentially reads the electrical signals generated by each pixel to obtain a second RAW image. The pixel arrangement in the second RAW image is consistent with that of the N-in-1 image sensor, and the resolution of the second RAW image is also consistent with that of the N-in-1 image sensor.

[0067] The following is combined with Figure 7 This plan will be further described below. Figure 7 This is a flowchart illustrating a shooting method provided in an embodiment of this application. The method includes:

[0068] S1, in response to the operation of opening the camera application, the phone displays the preview interface of the camera application, which displays the first preview image.

[0069] To open the camera app, you can tap the camera app icon on the phone's home screen.

[0070] Specifically, after receiving a command to open the camera app, the phone responds by launching the camera app and displaying its preview interface, which includes a first preview image. Generally, after launching the camera app, it enters the preview scene. For example, after launching the camera app, the camera activates, and the N-in-1 image sensor in the camera outputs a first RAW image in the first mode. The phone can then process the first RAW image into a first preview image using its ISP, allowing the phone to display a preview interface that includes this first preview image.

[0071] The processing can include a first processing, a second processing, and other preset processing to process the first RAW image into a first preview image. The ISP can include an IFE module and an IPE module. The IFE module can perform the first processing on the first RAW image, and the IPE module can perform the second processing on the first RAW image processed by the IFE module. For example, the IFE module can receive the RAW image output from the image sensor and perform the first processing. The first processing can include 3A processing. The IPE module can receive the RAW image output from the IFE module and perform the second processing on the RAW image. The second processing can include demosaicing. Demosaicing can convert a RAW image in the RAW domain into an image in the red, green, blue, RGB domain. For example, in a RAW image, a pixel only includes the value of one channel, such as the pixel value of pixel R being the value of the R channel, the pixel value of pixel G being the value of the G channel, and the pixel value of pixel B being the value of the B channel. Demosaicing can reconstruct the values ​​of each pixel in the R, G, and B channels based on these monochrome pixel values ​​using an interpolation algorithm, thereby generating a color image. After demosaicing, the IPE module can further perform other preset processing on the demosaiced image. These other preset processing include, but are not limited to, white balance and color correction.

[0072] After processing the first RAW image as described above, a first preview image can be obtained. The phone can then display this first preview image on the camera app's preview screen. For example, after the ISP obtains the first preview image, it returns it to the camera app, which then uses the CPU or GPU to render a preview screen including the first preview image. Alternatively, after obtaining the first preview image, the ISP uses the CPU or GPU to render a preview screen including the first preview image. The display then shows the rendered preview screen including the first preview image based on a preset screen refresh rate.

[0073] Understandably, after the camera app launches and enters the preview scene, the N-in-1 image sensor in the camera can output a first RAW image at a preset frequency in the first mode. For each frame of the first RAW image, the ISP can process it into a corresponding first preview image. Thus, after launching the camera app and entering the preview scene, the phone can display the camera app's preview interface, and the first preview image displayed on this interface is obtained based on the first RAW image output by the N-in-1 image sensor in the first mode.

[0074] In the preview scene, the first preview image displayed on the preview interface is obtained from the first RAW image output by the N-in-1 image sensor in the first mode. In the shooting scene, the phone can continue to display a preview image on the preview interface, referred to as the second preview image. The second preview image displayed on the preview interface is obtained from the second RAW image output by the N-in-1 image sensor in the second mode. For example, the phone can execute S2-S5 to display the second preview image on the camera application's preview interface in the shooting scene.

[0075] S2, the mobile phone receives the photo-taking operation and, under the condition that the preset conditions are met, controls the N-in-one image sensor to output M frames of the second RAW image in the second mode.

[0076] The mobile phone can receive the user's photo-taking operation. If the preset conditions are met, it indicates that a high-resolution image is required. At this time, the mobile phone can control the N-in-one image sensor to output a second RAW image in the second mode at the same preset frequency.

[0077] The preset conditions may include: a zoom magnification greater than 1. To obtain a high-resolution zoom image, the phone typically needs to first acquire a high-resolution image, and then crop the zoom image from the high-resolution image. Therefore, when the current zoom magnification is greater than 1, it is necessary to control the N-in-1 image sensor to output a second RAW image.

[0078] When a phone receives a photo-taking request, it first obtains the current zoom level. Then, based on this zoom level, the phone can determine whether preset conditions are met. For example, the phone can detect a user's zoom operation in the camera app's preview interface and determine the current zoom level accordingly.

[0079] For example, such as Figure 8 The camera application preview interface shown in (a) includes zoom buttons 901 that correspond one-to-one with multiple zoom levels, such as zoom button 1x for 1x zoom, zoom button 2x for 2x zoom, zoom button 3x for 3x zoom, and zoom button 5x for 5x zoom. Zooming can be an operation where the user selects one of the multiple zoom buttons, such as clicking a zoom button. The phone can obtain the zoom level corresponding to a zoom button based on the user's operation on that zoom button in the camera application preview interface; this obtained zoom level is the current zoom level.

[0080] For example, such as Figure 8As shown in (b), the camera application's preview interface includes a ruler 101 with an indicator arrow 102 indicating the current zoom level. The user can drag the indicator arrow 102 on the ruler 101 in the preview interface to select the zoom level. The phone can receive the user's dragging action on the indicator arrow 102 and obtain the user-selected zoom level based on the zoom level indicated by the indicator arrow 102 at the end of the dragging action. The obtained user-selected zoom level is the current zoom level.

[0081] For example, zooming can be a user's operation of scaling the first preview image in the preview interface. For instance... Figure 8 As shown in (c), the zoom operation can be, for example, a user's two-finger outward swipe on the preview screen. In response to this zoom operation, the phone obtains the swipe distance of the user's two fingers in real time, and based on the mapping relationship between the swipe distance and the zoom level, obtains the zoom level selected by the user. The zoom level selected by the user is the current zoom level.

[0082] Optionally, during the zoom operation, the size of the first preview image displayed in the camera application's preview interface can change as the zoom level selected by the user changes.

[0083] Under certain preset conditions, the phone can control the N-in-1 image sensor to adjust its operating mode. As an example, after launching the camera app, it enters the preview scene. In the preview scene, the N-in-1 image sensor's operating mode is set to the first mode by default, outputting the first RAW image in this mode. If a high-resolution image is required for the shooting scene, the operating mode of the N-in-1 image sensor can be switched to the second mode. Optionally, under certain preset conditions in the shooting scene, the operating mode of the N-in-1 image sensor can be switched to the second mode.

[0084] For example, when a mobile phone receives a photo-taking command, it sends a first switching command to the N-in-one image sensor. This first switching command instructs the N-in-one image sensor to switch its operating mode from a first mode to a second mode. In response to this first switching command, the N-in-one image sensor switches its operating mode from the first mode to the second mode.

[0085] Since current mobile phone built-in photography algorithms require at least one frame from the image sensor to capture an image, when the phone receives a photo-taking operation, it controls the N-in-one image sensor to output M frames of second RAW images in a second mode, provided preset conditions are met. Here, M is an integer greater than or equal to 1. For example, M can be 4, 6, 8, etc. The photography algorithm can include multi-frame noise reduction algorithms and multi-frame synthesis algorithms. For example, a multi-frame noise reduction algorithm can identify noise in an image based on information from multiple consecutive frames and perform noise reduction processing. A multi-frame synthesis algorithm can combine at least two images into a single high-quality image to improve image sharpness and dynamic range. This algorithm can eliminate blur and jitter in the image while preserving more detail and color information.

[0086] Optionally, after acquiring M second RAW images output by the N-in-1 image sensor, the mobile phone can switch the mode of the N-in-1 image sensor from the second mode to the first mode. That is, after obtaining the M second RAW images used to generate the captured image, i.e., after the camera application switches from the shooting scene to the preview scene, the N-in-1 image sensor continues to output first RAW images in the first mode for preview display. In this way, after the camera application switches from the shooting scene to the preview scene, the mobile phone can continue to preview based on the first RAW images output by the N-in-1 image sensor.

[0087] For example, after acquiring M second RAW images output by the N-in-1 image sensor, the mobile phone sends a second switching command to the N-in-1 image sensor. This second switching command instructs the N-in-1 image sensor to switch its operating mode from the second mode to the first mode. In response to this first switching command, the N-in-1 image sensor switches its operating mode from the second mode back to the first mode.

[0088] Optionally, S2 can also be: when the mobile phone receives a photo-taking operation, it controls the N-in-one image sensor to output M frames of second RAW images in the second mode. In other words, as long as the mobile phone receives a photo-taking operation, it can control the N-in-one image sensor to output M frames of second RAW images in the second mode, obtain and display a second preview image based on the M frames of second RAW images, and obtain and save the shooting operation based on the M frames of second RAW images.

[0089] For each frame of the M-frame second RAW image, the phone can execute S3–S5 to process the second RAW image into a second preview image and display the second preview image. For the M-frame second RAW image, the phone can execute S6 to process the M-frame second RAW image into a captured image and save the captured image. The following describes the different scenarios.

[0090] S3, the phone processes the second RAW image into a third RAW image.

[0091] The phone processes the second RAW image into a third RAW image. Specifically, the phone changes the pixel arrangement in the second RAW image from an N-pixel Bayer array to a Bayer array, resulting in the third RAW image. In other words, the pixel arrangement in the third RAW image is a Bayer array.

[0092] As mentioned earlier, the pixel arrangement in the second RAW image is an N-into-one Bayer array, not a Bayer array. However, the ISP in a mobile phone can only process RAW images with a Bayer array pixel arrangement. Therefore, it is necessary to convert the pixel arrangement in the second RAW image to a Bayer array. Although the remosaic algorithm can achieve this, it requires significant computing power to support complex interpolation and optimization processes, resulting in slower processing speeds, especially when processing high-resolution images like the second RAW image. Using the remosaic algorithm might cause the phone to fail to display the preview image according to its refresh rate, leading to severe lag in the preview interface. For example, processing M frames of images typically results in a lag of M*33ms in the preview interface, generally several hundred milliseconds. Therefore, this solution can conveniently convert the pixel arrangement in the second RAW image to a Bayer array through target processing. Below are two methods for mobile phones to process the second RAW image into a third RAW image.

[0093] In the first method, the target processing can modify the pixel arrangement of the second RAW image to a Bayer array, and the resolution of the processed second RAW image (i.e., the third RAW image) is 1 / N of the resolution of the unprocessed second RAW image.

[0094] Target processing can map n×n adjacent pixels of the same color in the second RAW image to a single pixel. Taking a quad RAW image as an example, as follows... Figure 9 As shown, the mobile phone, such as the ISP in the mobile phone, sequentially maps the 2×2 red pixels included in pixel region E of the second RAW image to pixel R; maps the 2×2 green pixels included in pixel region F of the second RAW image to pixel G; maps the 2×2 green pixels included in pixel region G of the second RAW image to pixel G; and maps the 2×2 blue pixels included in pixel region K of the second RAW image to pixel B.

[0095] For example, a mobile phone might use the pixel value of any one of its n×n adjacent pixels of the same color as the pixel value of the mapped pixel. Figure 9For example, the phone sequentially takes the pixel value of any one of the 2×2 red pixels included in pixel region E of the second RAW image as the pixel value of pixel R; takes the pixel value of any one of the 2×2 green pixels included in pixel region F of the second RAW image as the pixel value of pixel G; takes the pixel value of any one of the 2×2 green pixels included in pixel region G of the second RAW image as the pixel value of pixel G; and takes the pixel value of any one of the 2×2 blue pixels included in pixel region K of the second RAW image as the pixel value of pixel B.

[0096] For example, a mobile phone performs target operations on the pixel values ​​of n×n pixels, and uses the pixel value after the target operation as the pixel value of a mapped pixel. Target operations can include: summation, averaging, weighted summation, etc. Let's say the target operation is summation. Figure 9 For example, the phone sequentially uses the sum of the pixel values ​​of the 2×2 red pixels included in pixel region E of the second RAW image as the pixel value of pixel R; the sum of the pixel values ​​of the 2×2 green pixels included in pixel region F of the second RAW image as the pixel value of pixel G; the sum of the pixel values ​​of the 2×2 green pixels included in pixel region G of the second RAW image as the pixel value of pixel G; and the sum of the pixel values ​​of the 2×2 blue pixels included in pixel region K of the second RAW image as the pixel value of pixel B.

[0097] For example, a mobile phone processes a first RAW image into a third RAW image using a first neural network model. That is, the target processing can be inputting the first RAW image into the first neural network model. The input to the first neural network model can be the first RAW image, and the output of the first neural network model can be the third RAW image. The first neural network model is used to modify the pixel arrangement in the RAW image from an N-pixel array to a Bayer array. The first neural network model has the ability to output an RAW image based on the input RAW image. Specifically, the pixel arrangement in the input RAW image is an N-pixel array, and the pixel arrangement in the output RAW image is a Bayer array, with the resolution of the output RAW image being 1 / N of the resolution of the input RAW image. In other words, the first neural network model has the ability to process a RAW image with an N-pixel array as its pixel array into a RAW image with a standard Bayer array pixel array.

[0098] The following is a brief introduction to the sample construction and training process of the first neural network model.

[0099] Sample Construction: In the test environment, the same N-in-1 image sensor was used to take two shots of the same subject in the same shooting environment. In one shot, the N-in-1 image sensor was controlled to output RAW image A in the first mode, and in the other shot, it was controlled to output RAW image B in the second mode. RAW image A uses a Bayer array pixel arrangement, and its resolution is 1 / N of the N-in-1 image sensor's resolution. RAW image B uses an N-in-1 Bayer array pixel arrangement, and its resolution is the same as the N-in-1 image sensor's resolution. These two RAW images, A and B, constitute a training sample. By changing the shooting environment or subject and repeating the above steps, multiple training samples can be obtained.

[0100] Training Process: For a training sample, the RAW image B from the training sample is input into the first neural network model, which outputs an image. The loss between this image and the RAW image A is calculated. A smaller loss indicates a smaller difference between the two images, while a larger loss indicates a larger difference. Based on the loss, the model parameters of the first neural network model are optimized until the loss obtained based on a training sample is less than a threshold. The trained first neural network model has the ability to generate an output RAW image with a Bayer array pixel arrangement, given an input RAW image with an N-pixel grid. The resolution of the output RAW image is 1 / N of the resolution of the input RAW image.

[0101] In this embodiment, the resolution of the third RAW image after target processing is consistent with the resolution of the image corresponding to the zoom factor of the lossless zoom image obtained from the N-in-one image sensor. For example, if the zoom factor of the lossless zoom image obtained from the 4-in-one image sensor is 2x, the resolution of the image displayed on the screen at the 2x zoom factor is 1 / 4 of the resolution of the 4-in-one image sensor. As another example, if the zoom factor of the lossless zoom image obtained from the 9-in-one image sensor is 3x, the resolution of the image displayed on the screen at the 3x zoom factor is 1 / 9 of the resolution of the 9-in-one image sensor. The zoom factor of the lossless zoom image obtained from the 16-in-one image sensor is 4x. For example, at the 4x zoom factor, the resolution of the image displayed on the screen is 1 / 16 of the resolution of the 16-in-one image sensor. In other words, lossless zoom images from the N-in-one image sensor can be obtained through target processing.

[0102] The second method involves modifying the pixel arrangement of the second RAW image to a Bayer array, ensuring that the resolution of the processed second RAW image (i.e., the third RAW image) is identical to that of the original second RAW image. This second method can achieve the same effect as either hardware or software remosaic processing.

[0103] For example, a mobile phone uses a second neural network model to process a first RAW image into a third RAW image. That is, the target processing can be inputting the first RAW image into the second neural network model. The input to the second neural network model can be the first RAW image, and the output of the second neural network model can be the third RAW image. The second neural network model is used to modify the pixel arrangement in the RAW image from an N-in-one Bayer array to a Bayer array. The second neural network model has the ability to output a RAW image based on the input RAW image, where the pixel arrangement in the input RAW image is an N-in-one Bayer array, the pixel arrangement in the output RAW image is a Bayer array, and the resolution of the output RAW image is consistent with the resolution of the input RAW image, both consistent with the resolution of the N-in-one RAW image sensor. The sample construction and training process of the second neural network model is briefly introduced below.

[0104] Sample Construction: In the test environment, the N-in-1 image sensor was used to take two shots of the same subject in the same shooting environment. In one shot, the N-in-1 image sensor was controlled to output RAW image B in the second mode, and in the other shot, it was controlled to output RAW image C in the third mode. The electronic device including this N-in-1 image sensor has hardware remosaic processing capabilities. Thus, in the third mode, the pixel arrangement of the N-in-1 image sensor is switched from an N-in-1 Bayer array to a Bayer array through hardware remosaic processing. The N-in-1 image sensor can then output a second-format RAW image, such as RAW image C. The pixel arrangement in RAW image C is a Bayer array, and the resolution of RAW image C is consistent with the resolution of the N-in-1 image sensor. RAW images B and C are obtained, and these two images constitute a training sample. By changing the shooting environment or the subject and repeating the above actions, multiple training samples can be obtained.

[0105] Training Process: For a training sample, the RAW image B from the training sample is input into the second neural network model, which outputs an image. The loss between this image and the RAW image C is calculated. A smaller loss indicates less difference between the two images, while a larger loss indicates greater difference. Based on the loss, the parameters of the second neural network model are optimized until the loss obtained based on a training sample is less than a threshold. The trained second neural network model has the ability to generate an output RAW image with a Bayer array pixel arrangement from an input RAW image with an N-pixel array, and the resolution of the output RAW image is consistent with the resolution of the input RAW image. That is, the second neural network model has the ability to process a RAW image with an N-Bayer pixel array into a RAW image with a standard Bayer array pixel array.

[0106] In other words, the mobile phone can use a neural network model to process the second RAW image into a third RAW image. This neural network model can be either a first neural network model or a second neural network model.

[0107] Optionally, the mobile phone can first perform a first processing on the second RAW image output by the multi-image sensor, and then process the second RAW image after the first processing into a third RAW image. For example, the IFE module in the ISP of the mobile phone can first perform a first processing on the second RAW image output by the multi-image sensor, and then the first module in the ISP of the mobile phone can process the second RAW image after the first processing into a third RAW image.

[0108] After obtaining the third RAW image with pixels arranged in a Bayer array, the mobile phone can process the third RAW image into a second preview image. For example, the IPE module of the mobile phone's ISP can process the third RAW image into a second preview image. Exemplarily, the mobile phone can execute S4.

[0109] S4, the phone processes the third RAW image into a second preview image.

[0110] The pixels in the third RAW image are arranged in a Bayer array. The phone processes the third RAW image sequentially through demosaicing, preset processing, and zoom processing to obtain the second preview image. For example, the IPE module in the phone's ISP can perform demosaicing, preset processing, and zoom processing on the third RAW image. Demosaicing converts the RAW image in the RAW domain into an image in the red, green, blue, and RGB domains. For example, in a RAW image, each pixel includes only one channel value; pixel R's pixel value is the R channel value, pixel G's pixel value is the G channel value, and pixel B's pixel value is the B channel value. Demosaicing reconstructs the R, G, and B channel values ​​of each pixel based on these monochrome pixel values ​​using an interpolation algorithm, thereby generating a color image.

[0111] Preset processing includes, but is not limited to, noise reduction, automatic white balance (AWB), dynamic range control (DRC), color correction, gamma correction, and RGB2YUV (RGB to YUV conversion). The image after de-pixelation and preset processing will be referred to as the first image below.

[0112] Zoom processing is used to obtain an image that matches the zoom level selected by the user.

[0113] For example, when the phone obtains the third RAW image using the first method, zoom processing is not mandatory. Specifically, if the zoom factor selected by the user is the same as the zoom factor used to obtain the lossless zoom image from the N-in-1 image sensor, no cropping is required to perform zoom processing on the first image. If the zoom factor selected by the user is not the same as the zoom factor used to obtain the lossless zoom image from the N-in-1 image sensor, the phone performs zoom processing on the first image based on the user-selected zoom factor. For example, the phone first determines the adjustment factor M based on the user-selected zoom factor. Then, the phone performs zoom processing on the first image based on the adjustment factor M. Here, the adjustment factor M = n / the user-selected zoom factor. That is, the resolution of the image obtained by zooming the first image using the adjustment factor M is the same as the resolution of the image obtained by zooming the full-size image using the user-selected zoom factor. For example, if the user selects a zoom factor of 4, then the adjustment factor M = 2 / 4. The resolution of the 4-in-1 image sensor is 8000×6000. The resolution of the first image is 4000×3000. Therefore, the resolution of the image corresponding to the zoom level selected by the user is 4000×2 / 4×3000×2 / 4=2000×1500.

[0114] For example, when the phone obtains the first image using the second method, since the resolution of the first image is the same as that of the N-in-1 image sensor, regardless of whether the zoom factor selected by the user is the same as the zoom factor used to obtain the lossless zoom image from the N-in-1 image sensor, the first image needs to be cropped according to the user-selected zoom factor. For instance, the phone first uses the reciprocal of the user-selected zoom factor as the adjustment factor M. Then, the phone zooms the first image according to the adjustment factor M. For example, if the user selects a zoom factor of 2, the corresponding adjustment factor M is 1 / 2. The resolution of the 4-in-1 image sensor is 8000×6000. The resolution of the first image is 8000×6000. The resolution of the image corresponding to the user-selected zoom factor is 8000×1 / 2×60000×1 / 2=4000×3000. As another example, if the user selects a zoom factor of 4, the corresponding adjustment factor M is 1 / 4. The resolution of the first image is 8000×6000. The resolution of the image corresponding to the zoom level selected by the user is 8000×1 / 4×60000×1 / 4=2000×1500.

[0115] In this way, the second preview image is the zoomed image, and the phone can display the zoom effect to the user.

[0116] The S5 displays a second preview image in the camera app's preview interface.

[0117] For example, a mobile phone can render and composite a preview interface including a second preview image, and display the preview interface including the second preview image on the screen. For example, the ISP in the phone can trigger the CPU or GPU to render and composite a preview interface including the second preview image. As another example, the ISP in the phone can send the second preview image to the camera application, which can trigger the CPU or GPU to render and composite a preview interface including the second preview image. The CPU or GPU can store the rendered and composited preview interface in a preset buffer. The phone's screen reads the preview interface from the preset buffer based on a preset refresh rate and displays it.

[0118] S6: The phone obtains and saves the captured image based on the second RAW image of the M frame.

[0119] For example, a mobile phone can use a photography algorithm to process the second RAW image of the M-frame to obtain an intermediate image, and then perform zoom processing on the intermediate image to obtain the captured image and save it.

[0120] When the camera algorithm supports processing images with an N-in-1 Bayer array pixel arrangement, the phone directly inputs the second RAW image (M frames) into the algorithm to obtain an intermediate image. This intermediate image has the same resolution as the N-in-1 image sensor, making it a high-resolution image. The phone then zooms on the intermediate image according to the user-selected zoom level to obtain the captured image.

[0121] If the image processing algorithm does not support processing images with an N-in-1 Bayer array pixel arrangement, the phone can first process each of the M second RAW images into a fourth RAW image. Since image processing has relatively low latency requirements, the phone can use the remosaic algorithm to process the M second RAW images into M fourth RAW images. In these fourth RAW images, the pixels are arranged in a Bayer array, and the resolution of the fourth RAW image matches the resolution of the N-in-1 image sensor. For example, the fourth RAW image could be... Figure 3 As shown in (b) above. Next, the phone inputs the fourth RAW image of frame M into the image-taking algorithm to obtain an intermediate image. The resolution of this intermediate image is consistent with the resolution of the N-in-one image sensor. Then, the phone zooms the intermediate image according to the zoom level selected by the user to obtain the captured image.

[0122] Thus, in preset scenarios such as zoom shooting scenarios, in response to the shooting operation, the mobile phone can continue to display the preview image on the preview interface while simultaneously capturing a high-resolution shooting image. Furthermore, as mentioned above, the target processing method used in this application embodiment can conveniently and quickly process the second RAW image generated by the image sensor into a third RAW image, and generate the preview image and the shooting image based on the third RAW image. This meets the requirement for real-time display of the preview stream and enables the preview image to be displayed on the preview interface during the shooting scenario.

[0123] Optional, Figure 7 The method further includes: when the mobile phone receives a photo-taking operation, if preset conditions are not met, controlling the N-in-one image sensor to output N frames of first RAW images in a first mode. Here, N is an integer greater than or equal to 1. The mobile phone obtains a third preview image based on the N frames of first RAW images. For example, the mobile phone performs a first processing and a second processing on the first RAW images to obtain the third preview image, which is then displayed on the preview interface of the camera application. The first and second processing are detailed above and will not be repeated here. The mobile phone obtains and saves the corresponding captured image based on the N frames of first RAW images. For example, the mobile phone inputs the N frames of first RAW images into the photo-taking algorithm to obtain the captured image.

[0124] The preceding text, with reference to the accompanying drawings, described a method for capturing images in zoom scenarios. The shooting method provided in this application is applicable not only to zoom scenarios but also to other scenarios. Specifically, these other scenarios may be those where the N-in-1 image sensor needs to output a full-size image. Here, a full-size image is an image with the same resolution as the N-in-1 image sensor, and a high-resolution image. That is, these other scenarios may specifically be those where the N-in-1 image sensor needs to output a high-resolution image. When the mobile phone does not have hardware remosaic processing capabilities or does not use hardware remosaic processing capabilities, the full-size image output by the N-in-1 image sensor is the second RAW image mentioned earlier.

[0125] Other scenarios may include: high-brightness shooting scenarios, high dynamic range (HDR) mode enabled on the phone, high-resolution image output enabled on the phone, and shooting in professional mode. These will be described in detail below.

[0126] In high-brightness shooting scenarios, to capture high-resolution images, the N-in-1 image sensor in the phone can output a second RAW image. Optionally, the preset condition mentioned above can be: the current ambient light brightness is greater than a threshold. Here, the current ambient light brightness being greater than the threshold indicates that the current shooting scene is a high-brightness shooting scene. To capture a high-resolution image, in response to the photo-taking operation, the camera application enters shooting mode, and the N-in-1 image sensor can output M frames of second RAW images in the high-brightness shooting environment. For each frame of the M second RAW images, the phone processes the second RAW image into a third RAW image, and then processes the third RAW image into a second preview image. This processing does not include cropping. The phone can then display the second preview image on the preview screen. The phone inputs the M frames of second RAW images into the photo-taking algorithm to obtain the captured image. Alternatively, the phone processes the M second RAW images into a fourth RAW image, and then inputs the M frames of the fourth RAW image into the photo-taking algorithm to obtain the captured image. The resolution of the captured image is consistent with the resolution of the N-in-1 image sensor. In other words, in high-brightness shooting environments, the way the phone generates the second preview image and the captured image is similar to how it generates them in zoom scenarios. The difference is that in high-brightness shooting environments, the phone does not need to crop the third RAW image or the intermediate image output by the shooting algorithm; this intermediate image is the captured image. This allows users to capture high-resolution images in high-brightness shooting environments.

[0127] The phone can obtain the current ambient light brightness through an ambient light sensor. Optionally, as mentioned earlier, after the camera application is launched, the N-in-one image sensor in the camera can output a first RAW image at a preset frequency. The phone can obtain the current ambient light brightness based on the first RAW image output by the N-in-one image sensor. For example, the phone can analyze the brightness information of this first RAW image to obtain the current ambient light brightness.

[0128] When the phone's HDR function is enabled, the phone can output HDR images. HDR images expand the brightness range of an image compared to standard dynamic range (SDR) images, thus recording a larger range of brightness information and revealing more details in both bright and dark areas. Optionally, the preset condition mentioned above can be that the phone's HDR function is enabled. When the phone's HDR function is enabled, the phone can use the same method as when shooting images in bright shooting scenarios, which will not be elaborated here.

[0129] Optionally, the above preset condition can be that the phone's high-resolution image output function is enabled. When the phone's high-resolution image output function is enabled, the phone can use the same method as when shooting images in bright shooting scenes to capture images, which will not be elaborated here.

[0130] Optionally, the above preset condition can be that the shooting mode is set to professional mode. When the shooting mode is set to professional mode, the phone can take pictures using the same method as when shooting images in bright shooting scenarios, which will not be elaborated here.

[0131] As can be seen, in the preset scenario, the embodiments of this application are able to acquire high-resolution images while displaying preview images.

[0132] The preceding text, in conjunction with the accompanying drawings, described a shooting method provided by an embodiment of this application. The following text further describes this solution from the perspective of the interaction between various software / hardware modules within the mobile phone.

[0133] A mobile phone includes a camera application, an image sensor, and an ISP. The image sensor is the N-in-1 image sensor mentioned earlier. The ISP includes an IFE module, an IPE module, and a first module. The first module performs target processing on the second RAW image. The IFE module performs the first processing on the image. The IPE module performs the second processing on the image.

[0134] Taking a 4-in-1 image sensor as an example, combined with Figure 10 This plan will now be introduced.

[0135] Figure 10 The methods shown include:

[0136] S101, 4-in-1 image sensor startup.

[0137] When the camera application starts, the 4-in-1 image sensor can start simultaneously. The default operating mode of the 4-in-1 image sensor can be the first mode. After the 4-in-1 image sensor starts, it outputs the first RAW image in the first mode.

[0138] The S102, 4-in-1 image sensor sends the first RAW image to the IFE module.

[0139] In the first mode, the 4-in-1 image sensor reads out the electrical signals of 2×2 pixels of the same color as the electrical signal of a single pixel, obtaining a first RAW image. The resolution of the first RAW image is lower than the resolution of the 4-in-1 image sensor.

[0140] After acquiring the first RAW image, the 4-in-1 image sensor sends the first RAW image to the IFE module in the ISP.

[0141] S103, the IFE module performs the first processing on the first RAW image.

[0142] The first processing includes, but is not limited to, 3A processing.

[0143] S104, the IFE module sends the processed first RAW image to the IPE module.

[0144] S105, the IPE module performs a second processing on the processed first RAW image.

[0145] The second processing step can be de-mosaicing. De-mosaicing is used to obtain the values ​​of pixels in the R, G, and B channels of the image. Optionally, IPE can also perform other processing on the processed first RAW image, such as balancing or color correction. Optionally, if the zoom level is greater than 1 (a preset condition), IPE will also crop the processed first RAW image.

[0146] The first RAW image processed by the IPE module can be the first preview image.

[0147] S106, the IPE module sends the first preview image to the camera application.

[0148] S107, the camera app displays the first preview image.

[0149] Specifically, the camera application can call the CPU or GPU to render a preview interface, including the first preview image, and display that preview interface.

[0150] S108, in response to a photo-taking operation, under preset conditions, the camera application sends a first switching command to the 4-in-1 image sensor.

[0151] The first switching command is used to switch the operating mode of the 4-in-1 image sensor from the first mode to the second mode. In the second mode, the 4-in-1 image sensor reads the electrical signals of each pixel to obtain a second RAW image. The resolution of the second RAW image is equal to the resolution of the 4-in-1 image sensor.

[0152] S109, in response to the first switching command, the 4-in-1 image sensor sends the second RAW image to the IFE module.

[0153] In response to the first switching command, the 4-in-1 image sensor switches its operating mode from the first mode to the second mode. After switching modes, the 4-in-1 image sensor can acquire a second RAW image. After acquiring each frame of the second RAW image, the 4-in-1 image sensor sends the second RAW image to the IFE module in the ISP.

[0154] S110, the IFE module performs the first processing on the second RAW image.

[0155] The first processing includes, but is not limited to, 3A processing.

[0156] S111, the IFE module sends the processed second RAW image to the first module.

[0157] S112, the first module performs target processing on the processed second RAW image.

[0158] The first module is used to perform target processing on the second RAW image to transform it into a third RAW image. The target processing process is described in detail above and will not be repeated here.

[0159] S113, the first module sends the third RAW image to the IPE module.

[0160] S114, the IPE module performs a second processing on the third RAW image.

[0161] The second processing step can be de-mosaicing. De-mosaicing is used to obtain the values ​​of pixels in the R, G, and B channels of the image. Optionally, IPE can also perform other processing on the third RAW image, such as balancing and color correction. Optionally, if the zoom level is greater than 1 (a preset condition), IPE will also crop the third RAW image.

[0162] The third RAW image processed by the IPE module can be used as a second preview image.

[0163] S115, the IPE module sends a second preview image to the camera application.

[0164] S116, the camera app displays a second preview image.

[0165] S117, the camera application sends a second switching command to the 4-in-1 image sensor.

[0166] The first switching command is used to switch the operating mode of the 4-in-1 image sensor from the second mode to the first mode. For example, after a preset duration, the camera sends a second switching command to the 4-in-1 image sensor. Within the preset duration, the 4-in-1 image sensor can output M frames of second RAW images.

[0167] Optionally, the IFE module can also send the second RAW image of M after the first processing to the camera application, which can then call the image capture algorithm to obtain the captured image.

[0168] This application provides an electronic device including a memory, a display screen, and one or more processors. The display screen is coupled to the processors. The memory stores computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the mobile phone in the above method embodiments. The structure of the electronic device can be referred to... Figure 5 The structure of the electronic device 500 shown.

[0169] This application embodiment also provides a computer storage medium, which includes computer instructions, when the computer instructions are executed in the aforementioned electronic device (such as...). Figure 5 When the electronic device 500 shown is run, it causes the electronic device to perform the various functions or steps in the above method embodiments.

[0170] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps described in the above method embodiments.

[0171] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0174] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A shooting method, characterized in that, Applied to an electronic device, the electronic device including an image sensor, the pixel array of the image sensor being an N Bayer pixel array, where N is an integer greater than 1, the method includes: Displays a preview of the camera application; A first preview image is displayed on the preview interface based on the first RAW image output by the image sensor; the pixel array of the first RAW image is a standard Bayer array; the resolution of the first RAW image is 1 / N of the resolution of the image sensor. In response to the received photo capture operation, a second preview image is displayed on the preview interface based on the second RAW image output by the image sensor, and the captured image is saved; the pixel array of the second RAW image is an N-bay pixel array.

2. The method according to claim 1, characterized in that, The second RAW image is the output of the image sensor in the second mode; Before displaying the second preview image on the preview interface based on the second RAW image generated by the image sensor, the method further includes: Under preset conditions, the operating mode of the image sensor is switched from a first mode to a second mode; in the first mode, the image sensor outputs a RAW image with a standard Bayer pixel array and a resolution of 1 / N of the image sensor's resolution; in the second mode, the image sensor outputs a RAW image with an N Bayer pixel array; wherein, satisfying the preset conditions indicates that the electronic device needs to output a high-resolution captured image.

3. The method according to claim 1 or 2, characterized in that, The step of displaying the second RAW image generated by the image sensor as a second preview image on the preview interface includes: The pixel value of any one of the N adjacent pixels of the same color in the second RAW image is used as the pixel value of a pixel in the third RAW image; The second preview image is obtained based on the third RAW image and displayed on the preview interface.

4. The method according to claim 1 or 2, characterized in that, The step of displaying the second RAW image generated by the image sensor as a second preview image on the preview interface includes: A target operation is performed on the pixel values ​​of N adjacent pixels of the same color in the second RAW image, and the pixel value after the target operation is used as the pixel value of a pixel in the third RAW image; the target operation is any one of summation, weighted summation, or mean calculation; The second preview image is obtained based on the third RAW image and displayed on the preview interface.

5. The method according to claim 1 or 2, characterized in that, The step of displaying the second RAW image generated by the image sensor as a second preview image on the preview interface includes: The second RAW image is input into the neural network model to obtain the third RAW image; the neural network model has the ability to process a RAW image with an N Bayer pixel array into a RAW image with a standard Bayer pixel array. The second preview image is obtained based on the third RAW image and displayed on the preview interface.

6. The method according to any one of claims 2-5, characterized in that, The captured image is obtained from M second RAW images, where M is an integer greater than or equal to 1. The method further includes: After obtaining the M second RAW images, the operating mode of the image sensor is switched from the second mode to the first mode, so that the RAW images output by the image sensor in the first mode can continue to be displayed on the preview interface.

7. An electronic device, characterized in that, The electronic device includes: a memory, a camera, and one or more processors; the camera, the memory, and the processors are coupled; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1-6.

Images