Photography methods and electronic devices
By dividing the image processing during the photo-taking process into first and second stages, and determining whether to execute the second stage processing based on system resources and user operation conditions, the problem of the second stage processing affecting the smoothness of the foreground application is solved, achieving efficient image quality and user experience.
Patent Information
- Application Number
- CN202410983219.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-07-19
AI Technical Summary
The second stage of image processing requires a large amount of hardware resources during the photo-taking process, which affects the smoothness of other foreground applications and causes overheating and lag issues.
The first and second stages of image processing are separated. The raw image information captured by the camera is displayed and stored in real time. The system determines whether to execute the second stage of image processing based on system resources and user operation conditions, making full use of the idle resources of electronic devices.
This avoids the impact of the second-stage image processing on the smoothness of other foreground applications, improves user experience and device performance, and ensures that high-quality image processing is completed quickly under suitable conditions.
Smart Images

Figure CN121418658B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of photography, and more particularly to a method for taking photographs and an electronic device. Background Technology
[0002] To ensure smooth photo taking when using a camera app on an electronic device, the entire process can be divided into two stages: In the first stage, image processing, after the user clicks the shutter button, the camera app performs lightweight image processing on the raw image data captured by the camera to output a thumbnail for quick display. Then, in the second stage, the background processes the raw image data captured by the camera to perform heavyweight image processing to output the target image.
[0003] The second stage of image processing requires a large amount of hardware resources, which can affect the smoothness of other foreground applications and cause overheating and lag issues. Summary of the Invention
[0004] This application provides a photo-taking method and an electronic device to avoid the second-stage image processing affecting the smoothness of other foreground applications.
[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:
[0006] In a first aspect, a method for taking photos is provided, applied to an electronic device. The method includes: launching a camera application and displaying an image captured by the camera in real time; in response to a user's shooting operation, acquiring and storing the original image information of the image captured by the camera, and performing a first-stage image processing on the image to obtain a thumbnail; and in response to at least one of the following information, performing a second-stage image processing on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient.
[0007] The photographing method and electronic device provided in this application perform a first-stage image processing to obtain a thumbnail when taking a picture. The method determines whether to perform a second-stage image processing based on the original image information based on the following information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or a small window, whether the touch point is in the camera application, whether the user is interacting with the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient. This allows the first and second stages of image processing to run independently, and more strictly limits the triggering conditions for the second stage of image processing, preventing it from affecting the smoothness of other foreground applications.
[0008] In one possible implementation, in response to at least one of the following information, a second stage of image processing is performed on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or a small window, whether the touch point is on the camera application, whether the user is interacting with the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient. Specifically, if the electronic device has not triggered overheat protection, the camera application is open, the electronic device has enabled split-screen or a small window, the user's touch point is not on the camera application, the user is not interacting with the image, the electronic device is not in power-saving mode, and the electronic device has sufficient system resources, then the original image information is read, and the second stage of image processing is performed on the image based on the original image information. This fully utilizes the idle system resources of the electronic device to perform the second stage of image processing on the image.
[0009] In one possible implementation, in response to at least one of the following information, a second stage of image processing is performed on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or a small window, whether the touch point is in the camera application, whether the user is interacting with the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient. Specifically, if the electronic device has not triggered overheat protection, the camera application is not open, the user is not interacting with the image, the electronic device is not in power-saving mode, and the electronic device's system resources are sufficient, then the original image information is read, and the second stage of image processing is performed on the image based on the original image information. This fully utilizes the idle system resources of the electronic device to perform the second stage of image processing.
[0010] In one possible implementation, in response to at least one of the following information, a second stage of image processing is performed on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or a small window, whether the touch point is on the camera application, whether the user is manipulating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient. This includes: if the electronic device has not triggered overheat protection, the camera application is open, the electronic device has enabled split-screen or a small window, the user's touch point is not on the camera application, and the user is manipulating the image, then the original image information is read, and the second stage of image processing is performed on the image based on the original image information. This fully utilizes the idle system resources of the electronic device, and by quickly performing the second stage of image processing on the image while the user is manipulating it, it ensures that the user can obtain the target image in the shortest possible time.
[0011] In one possible implementation, in response to at least one of the following information, a second stage of image processing is performed on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or a small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient. This includes: if the electronic device has not triggered overheat protection, the camera application is not open, and the user is operating the image, then the original image information is read, and the second stage of image processing is performed on the image based on the original image information. This fully utilizes the idle system resources of the electronic device, and by quickly performing the second stage of image processing on the image while the user is operating it, it ensures that the user can obtain the target image in the shortest possible time.
[0012] In one possible implementation, the method further includes: if there are images that have not undergone second-stage image processing for a certain period of time, then the second-stage image processing is performed on these images based on their original image information to avoid the loss of these images.
[0013] In one possible implementation, the method further includes: if multiple images use the same image processing algorithm, then performing a second-stage image processing on multiple images in batches based on the original image information of the multiple images, in order to reduce the overhead of loading and initializing the same image processing algorithm.
[0014] In a second aspect, an electronic device is provided, including a camera, a display screen, a processor, and a memory. The processor controls the display screen to display images and controls the camera to capture images. The memory stores instructions, which, when executed by the processor, cause the electronic device to perform the method described in the first aspect and any embodiment thereof.
[0015] Thirdly, a computer-readable storage medium is provided that stores instructions which, when executed on an electronic device, cause the electronic device to perform the method as described in the first aspect and any embodiment thereof.
[0016] Fourthly, a computer program product is provided, including instructions that, when executed on the electronic device, cause the electronic device to perform the method as described in the first aspect and any embodiment thereof.
[0017] Fifthly, a chip system is provided, including a processor for supporting electronic devices in implementing the functions described in the first aspect above. In one possible design, the device further includes interface circuitry for receiving signals from other devices (e.g., memory) or sending signals to other devices (e.g., a communication interface). The chip system may include a chip and may also include other discrete devices.
[0018] The technical effects of the second to fifth aspects refer to the technical effects of the first aspect and any of its embodiments, and will not be repeated here. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the appearance of an electronic device provided in an embodiment of this application;
[0020] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0021] Figure 3 A schematic diagram of a software system running on an electronic device, provided as an embodiment of this application;
[0022] Figure 4 A flowchart illustrating a photographing method provided in an embodiment of this application;
[0023] Figure 5 This application provides an embodiment of a diagram illustrating how second-stage image processing affects the smoothness of the foreground application.
[0024] Figure 6 A schematic diagram illustrating how the failure of image processing in the second stage affects user experience, as provided in an embodiment of this application.
[0025] Figure 7 This is a schematic diagram illustrating how second-stage image processing affects the smoothness of the foreground application and how the failure to perform second-stage image processing affects the user experience, as provided in an embodiment of this application.
[0026] Figure 8 A flowchart illustrating another photographing method provided in an embodiment of this application;
[0027] Figure 9 A schematic diagram illustrating the launching of a camera application, provided as an embodiment of this application;
[0028] Figure 10 A schematic diagram illustrating the launching of a gallery application provided in an embodiment of this application;
[0029] Figure 11 A schematic diagram of a split-screen display interface provided in an embodiment of this application;
[0030] Figure 12 A schematic diagram of a display window provided in an embodiment of this application;
[0031] Figure 13 A schematic diagram illustrating the priority of a judgment condition provided in an embodiment of this application;
[0032] Figure 14 A flowchart illustrating another photographing method provided in this application embodiment;
[0033] Figure 15 A flowchart illustrating yet another photographing method provided in an embodiment of this application;
[0034] Figure 16 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation
[0035] First, some concepts involved in this application will be described.
[0036] The terms "first" and "second" used in the embodiments of this application are only used to distinguish features of the same type and should not be construed as indicating relative importance, quantity, order, etc.
[0037] The terms "exemplary" or "for example" used in the embodiments of this application are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0038] The terms "coupling" and "connection" used in the embodiments of this application should be interpreted broadly. For example, they can refer to a physical direct connection or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.
[0039] like Figure 1As shown, this application embodiment provides an electronic device 101, which can be mobile or fixed. The electronic device can be deployed on land (e.g., indoors or outdoors, handheld or vehicle-mounted), on water (e.g., a ship), or in the air (e.g., an airplane, a balloon). This electronic device can be referred to as user equipment (UE), access terminal, terminal unit, subscriber unit, terminal station, mobile station (MS), mobile station, terminal agent, or terminal device, etc. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, smart bracelet, smart screen, smartwatch, earphone, smart speaker, terminal in industrial control, terminal in self-driving, terminal in remote medical care, terminal in smart grid, terminal in transportation safety, terminal in smart city, terminal in smart home, etc. This application embodiment does not limit the specific type and structure of the electronic device. One possible structure of the electronic device is described below.
[0040] like Figure 2 As shown, taking a mobile phone as an example, electronic device 101 may include, at the hardware level, a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a power management module 240, a battery 241, a wireless charging coil 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. Optionally, in some embodiments, an audio digital signal processor (ADSP) 243 may also be included.
[0041] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 101. In other embodiments of this application, the electronic device 101 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0042] Processor 210 may include one or more processing units, such as: a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), an application processor (AP), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a baseband processor, and a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. For example, processor 210 may be an application processor (AP). Alternatively, processor 210 may be integrated into a system-on-chip (SoC). Or, processor 210 may be integrated into an integrated circuit (IC) chip. The processor 210 may include an analog front end (AFE) and a micro-controller unit (MCU) in an IC chip.
[0043] The ADSP 243 can be coupled to the audio module 270 and the sensor module 280. The ADSP 243 can process audio signals and sensor data. Even when the processor is in sleep mode, the ADSP 243 can remain operational, thereby reducing the power consumption of the electronic device.
[0044] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 101. In other embodiments of this application, the electronic device 101 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.
[0045] The external storage interface 220 can be used to connect an external memory card, such as a micro SanDisk (Micro SD) card, to expand the storage capacity of the electronic device 101. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.
[0046] Internal memory 221 can be used to store computer executable program code, which includes computer instructions. Processor 210 executes various functional applications and data processing of electronic device 101 by running the computer instructions stored in internal memory 221, such as executing the photographing method involved in the embodiments of this application. In addition, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.
[0047] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0048] Electronic device 101 can implement audio functions such as music playback and recording through audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, and application processor.
[0049] Audio module 270 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. In some embodiments, audio module 270 may be located in processor 210, or some functional modules of audio module 270 may be located in processor 210. Speaker 270A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Receiver 270B, also called a "handpiece," is used to convert audio electrical signals into sound signals. Microphone 270C, also called a "microphone" or "microphone," is used to convert sound signals into electrical signals. Electronic device 101 may be equipped with at least one microphone 270C. Headphone jack 270D is used to connect wired headphones. Headphone jack 270D may be a USB interface 230, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.
[0050] Buttons 290 include a power button, volume buttons, etc. Buttons 290 can be mechanical buttons or touch buttons. Electronic device 101 can receive button input and generate key signal inputs related to user settings and function control of electronic device 101. Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 292 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to achieve contact and separation with electronic device 101. Electronic device 101 can support one or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. In some embodiments, the electronic device 101 employs an embedded SIM (eSIM) card, which can be embedded in the electronic device 101 and cannot be separated from the electronic device 101.
[0051] Electronic device 101 can implement display functions through a GPU, display screen 294, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute computer instructions to generate or modify display information.
[0052] The sensor module 280 may include pressure sensors, gyroscopes or gravity sensors, barometric pressure sensors, magnetic field sensors, acceleration sensors, distance sensors, structured light sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, etc.
[0053] Battery 241 may include one or more batteries to power a load.
[0054] The power management module 240 is used to receive charging input from a charger. The charger can be a wireless charger, such as a wireless charging dock, or other electronic device 101 with reverse wireless charging capability. The power management module 240 can receive wireless charging input via the wireless charging coil 242 of the electronic device. The charger can also be a wired charger; for example, the power management module 240 can receive charging input from a wired charger via a USB interface 230. The power management module 240 is also referred to as a charging chip.
[0055] The power management module 240 charges the battery 241 while simultaneously supplying power to the electronic devices. It receives input from the battery 241 and powers the processor 210, internal memory 221, external memory interface 220, display screen 294, camera 293, and wireless communication module 260. The power management module 240 can also monitor parameters such as the battery 241's capacity, voltage, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 240 may also be integrated into the processor 210.
[0056] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. For example... Figure 1 As shown in Figure A, the plane on which the display screen 294 of the electronic device 101 is located is referred to as the front side. In some embodiments, the electronic device 101 may include one or more display screens 294.
[0057] Camera 293 is used to capture still images or moving videos (collectively referred to as images). In some embodiments, electronic device 101 may include one or N cameras 293, where N is a positive integer greater than 1. Camera 293 generally includes an optical lens and a photosensitive element, which can be any photosensitive device such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS). During image capture, reflected light from the object being photographed passes through the optical lens and is projected onto the photosensitive element. The photosensitive element converts the received light signal into an electrical signal, which is then processed by the ISP and sent to the processor 210, thus obtaining a frame of image. Camera 293 may include, for example, Figure 1 The front-facing camera 2931 shown in Figure A and as shown in Figure A Figure 1 The main camera 2932 shown in B refers to the front-facing camera, which means that the camera 293 and the display screen 294 are located on the same plane of the electronic device 101. The main camera means that the camera 293 and the display screen 294 are on different planes.
[0058] The processor 210 may also include a memory for storing computer instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store computer instructions or data that the processor 210 has just used or that are being used repeatedly. If the processor 210 needs to use the same computer instructions or data again, it can retrieve them directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves system efficiency.
[0059] In some embodiments, the processor 210 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.
[0060] The wireless communication function of electronic device 101 can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor.
[0061] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 101 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.
[0062] Mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G for use on electronic device 101. Wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), satellite communication (such as satellite data communication and satellite voice communication), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies for use on electronic device 101. For example, wireless communication module 260 may include a satellite communication chip to enable satellite communication. Furthermore, antenna 1 of electronic device 101 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling electronic device 101 to communicate with networks and other devices via wireless communication technology.
[0063] like Figure 3 As shown, taking the Android operating system running on an electronic device as an example, the software architecture of the processor 210 includes the application layer, framework layer, system runtime library layer, hardware abstraction layer (HAL) layer, and kernel layer.
[0064] The kernel layer is the layer between hardware and software. For example, the kernel layer includes display drivers, camera drivers, audio drivers, sensor drivers, etc. The display driver is used to drive the display screen to show images or receive user touch operations; the camera driver is used to drive the camera to capture images; the audio driver is used to drive the microphone to capture sound and the speaker to play sound; and the sensor driver is used to drive the ADSP to acquire sensor data from various sensors.
[0065] The Hardware Abstraction Layer (HAL) is used to abstract hardware. The HAL hides the hardware interface details of a specific platform, providing the operating system with a virtual hardware platform and exhibiting hardware independence. For example, the HAL includes display modules, camera modules, audio modules, and sensor modules. The display module provides a virtual display, the camera module provides a virtual camera, the audio module provides a virtual microphone and speaker, and the sensor module provides a virtual camera.
[0066] The system runtime library layer includes C / C++ libraries and runtime libraries. Many core components and services of the Android operating system are built from native code and require C / C++ libraries. When an application is first installed, it is pre-compiled into machine code form as a runtime library; this process is called pre-compilation. This allows for faster startup and execution of the application by running the machine code.
[0067] The framework layer provides the application programming interface (API) and programming framework for the applications in the application layer. The framework layer includes predefined functions, such as a window manager, content provider, view system, and notification manager. Specifically, the framework layer also includes an image processing module for executing the image capture method described in this application.
[0068] The application layer can include a series of application packages, such as desktop, camera, calling, game, short video applications, etc.
[0069] like Figure 4 As shown, when a user takes a photo using the camera application of an electronic device, to ensure smooth photo taking, the entire photo taking process can be divided into two stages: S001, after the camera application is opened and the user clicks the shutter button, the camera application performs lightweight image processing (e.g., image compression) on the raw image data captured by the camera to output a thumbnail (referred to as the first stage image processing in this embodiment), which is used for quick display. Then the camera application is closed. S002, a separate thread performs heavyweight image processing on the raw image data captured by the camera in the background to output a target image (referred to as the second stage image processing in this embodiment). The image quality of the target image is better than the thumbnail, for example, it is a high-definition image, and therefore can be used to replace the thumbnail.
[0070] The second stage of image processing requires the allocation of significant hardware resources such as CPU, GPU, and memory, which can affect the smoothness of camera applications and cause overheating and lag issues. For example... Figure 5As shown, if a user takes a photo and then operates the camera app until the camera app exits the foreground, the second-stage image processing will continue to run, affecting the operation of other applications in the foreground and causing lag in the foreground applications. This effect is particularly noticeable in high-load scenarios such as games. If the second-stage image processing runs simultaneously with the foreground camera app, its performance will be limited to ensure smooth camera shooting, thus reducing the image quality of the second-stage image processing.
[0071] One improvement is to separate the second-stage image processing from the first-stage image processing. That is, the first-stage image processing is performed when taking a photo using the camera app, outputting a thumbnail and storing the raw image data captured by the camera. The second-stage image processing is then performed when the user views, edits, or shares the image. However, this approach degrades the user experience. For example, ... Figure 6 As shown, if the second stage of image processing is still in progress when a user takes a photo and is viewing, editing, or sharing the image, the image the user sees is still a thumbnail with poor quality. Furthermore, waiting for the second stage of image processing to complete takes time, which may lead the user to delete the image due to dissatisfaction with the image quality and to question the camera's performance. Figure 7 As shown, if the second stage of image processing begins after the camera app exits the foreground, while ensuring smooth image capture, background processing creates resource contention with other foreground applications, impacting their user experience. Furthermore, in scenarios with split-screen or windowed camera displays, users may not close the camera app after taking a photo but instead directly view, edit, or share the image. In this case, the second stage of image processing hasn't been executed, resulting in poor image quality and negatively affecting the user experience.
[0072] Therefore, the image-taking method provided in this application separates the first-stage image processing from the second-stage image processing. After receiving a user's command to launch a camera application, the electronic device, in response to the user's command, launches the camera application, captures an image, and displays a preview interface in the camera application. After the camera application is launched, the electronic device receives the user's image-taking command, which can be a single shot, a quick shot, or a burst shot. In response to the image-taking command, the camera application performs the first-stage image processing, outputs a thumbnail, and stores the original image information of the image captured by the camera. When certain preset conditions are met, such as sufficient system resources, the second-stage image processing is then executed, thus not affecting the smoothness of other foreground applications.
[0073] Figure 8 This is a flowchart illustrating a photographing method provided in an embodiment of this application, as shown below. Figure 8 As shown, the method includes the following steps:
[0074] S101. Launch the camera application. The camera application will display images captured by the camera in real time.
[0075] After the electronic device detects that the user has launched the camera app, the camera app launches in response to that action. For example, Figure 9 As shown in Figure A, the electronic device detects the user's action of clicking the camera application icon 801, as follows: Figure 9 As shown in Figure B, in response to this operation, the camera application is launched, and the image captured by the camera is displayed in real time on the camera application's interface. For example, after the electronic device detects that the user says "Open camera application," it launches the camera application in response to this voice input and displays the image captured by the camera in real time. In this case, the image captured by the camera is not stored.
[0076] S102. In response to the user's shooting operation, the camera application acquires and stores the raw image information of the image captured by the camera, and performs the first stage of image processing on the image to obtain a thumbnail.
[0077] For example, such as Figure 9 As shown in Figure B, the camera application detected the user's action of clicking the shutter button 802, as follows: Figure 9 As shown in Figure C, in response to this operation, a first-stage image processing is performed on the image to obtain a thumbnail. This thumbnail is displayed in the preview frame 803 of the camera application, and the original image information of the image captured by the camera is stored. The camera application also sends storage indication information to the gallery application and the image processing module. The storage indication information is used to indicate that the original image information of the image captured by the camera has been stored.
[0078] Raw image information refers to all information related to the image captured by the camera. Raw image information may include, but is not limited to, at least one of the following: raw image data, image processing algorithms involved in the second-stage image processing, camera hardware parameters, camera software parameters, processor parameters, and environmental parameters.
[0079] The raw image data includes, but is not limited to, at least one of the following: pixel information (e.g., red, green, blue (RGB)) and image format (e.g., bitmap (BMP) or joint photo graphic experts group (JPEG)).
[0080] The image processing algorithms involved in the second stage of image processing (i.e., weighted image processing algorithms) include, but are not limited to, at least one of the following: image enhancement algorithms (e.g., histogram equalization algorithm, Laplacian sharpening algorithm, median filtering algorithm), image segmentation algorithms (e.g., threshold segmentation algorithm, region growing algorithm), image white balance processing algorithms, feature extraction algorithms (e.g., scale-invariant feature transform algorithm, oriented gradient histogram algorithm), image recognition and classification algorithms (e.g., convolutional neural network algorithm, support vector machine algorithm), image reconstruction algorithms (e.g., Fourier transform, wavelet transform), and computer vision algorithms (e.g., target tracking algorithm, 3D reconstruction algorithm).
[0081] The hardware parameters of a camera include, but are not limited to, at least one of the following: sensor type, sensor size, number of pixels, aperture size, lens type (e.g., prime lens, wide-angle lens, telephoto lens), shutter speed, and ISO range.
[0082] The camera's software parameters include, but are not limited to, at least one of the following: parameters used during white balance adjustment, parameters used during gamma correction, parameters used during color correction, parameters used during sharpening, exposure mode (e.g., automatic or manual exposure), focus mode (e.g., autofocus, manual focus, continuous autofocus), metering mode, resolution (image width and height, such as 3840*2160 pixels), compression ratio, and whether a flash is used.
[0083] Processor parameters include, but are not limited to, at least one of the following: number of processor cores and threads, number of graphics processor cores, and remaining memory.
[0084] Environmental parameters may include lighting conditions (e.g., natural light, artificial light, front lighting, backlighting), shooting time, weather information, and geographic location information (e.g., latitude and longitude, altitude of the shooting location).
[0085] After completing step S102, you can exit the camera application, meaning that the subsequent step S103 does not depend on the camera application being open.
[0086] S103. In response to at least one of the following information, the image processing module does not perform second-stage image processing on the image, or reads the original image information of the image and performs second-stage image processing on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power saving mode, and whether the system resources are sufficient.
[0087] Step S103 is not necessarily executed immediately after steps S101-S102. After steps S101-S102, if there are images that have not undergone the second stage of image processing, the image processing module can continue to run step S103 in the background. Alternatively, when a user manipulates images through a gallery application or a third-party application (such as a chat application), if there are images that have not undergone the second stage of image processing, the image processing module will execute step S103. User operations on images include, but are not limited to, viewing, sharing, editing, saving, and deleting.
[0088] For example, such as Figure 10 As shown in Figure A, the electronic device displays the desktop, responding to the user clicking the gallery application icon 901, as... Figure 10 As shown in Figure B, the electronic device displays a gallery application. In response to the user clicking the image icon 902 in the gallery application, as... Figure 10 As shown in Figure C, the electronic device executes step S103, performing a second-stage image processing on the image based on the original image information, and then displays the image after the second-stage image processing. Specifically, if the user's operation on the image is deletion, the original image information is deleted to avoid wasting storage resources. If the user's operation on the image is viewing, sharing, editing, or saving, the image processing module executes step S103, for example, finding the image in the processing queue based on its name, prioritizing the reading of the image's original image information, and performing the second-stage image processing on the image based on the original image information.
[0089] Overheat protection refers to the protection mechanism that if the power supply current of an electronic device exceeds a threshold, or if the temperature of the electronic device exceeds a threshold, the electronic device will overheat, reducing the user experience and even posing a risk of spontaneous combustion. Therefore, the electronic device will trigger overheat protection, reducing power consumption by shutting down services, reducing performance, or shutting down, thereby reducing the heat generated by the electronic device.
[0090] For example, such as Figure 11 As shown, when an electronic device activates split-screen mode, it displays a split-screen interface. This interface includes a first window 1001, a second window 1002, and a split-screen line 1003 between the two windows. The first window 1001 and the second window 1002 run different applications; for example, the first window 1001 runs a camera application, and the second window 1002 runs a call application. In the camera application, a touch point indicates that the system detects the user tapping any location within the camera application window with their finger.
[0091] For example, such as Figure 12As shown, when an electronic device opens a small window, it displays the desktop and at least one other window, such as a first window 1101 and a second window 1102. The first window 1101 and the second window 1102 run different applications; for example, the first window 1101 runs a camera application, and the second window 1102 runs a calling application. In the camera application, a touch point indicates that the system detects that the user has tapped any location within the window containing the camera application with their finger.
[0092] Power saving mode, also known as super power saving mode, reduces the power consumption of electronic devices by turning off services or reducing performance, thereby extending battery life.
[0093] System resources include, but are not limited to, at least one of the following: CPU power status indicator (PSI) and utilization, GPU PSI and utilization, memory PSI and utilization, input / output (IO) interface PSI and utilization, temperature, power consumption, and operating scenarios (e.g., high-load scenarios such as gaming, benchmarking, and interactive scenarios, as well as low-load scenarios such as web browsing, audio / video playback, and idle / non-operational scenarios).
[0094] For example, sufficient system resources include at least one of the following: CPU PSI is less than a threshold, CPU utilization is less than a threshold, GPU PSI is less than a threshold, GPU utilization is less than a threshold, memory PSI is less than a threshold, memory utilization is less than a threshold, I / O interface PSI is less than a threshold, I / O interface utilization is less than a threshold, electronic device temperature is less than a threshold, electronic device battery power is greater than a threshold, and electronic device is operating in a low-load scenario.
[0095] like Figure 13 As shown, the highest priority is whether the electronic device has triggered overheat protection. The priority decreases sequentially depending on whether the touch point is in the camera application and whether the user is interacting with the image. The priority of whether system resources are sufficient and whether the electronic device is in power-saving mode are the lowest, having the same priority. Whether the camera application is open and whether split-screen or a small window is enabled are all to help determine whether the touch point is in the camera application. This optimizes the user's photography experience and the overall electronic device usage experience.
[0096] like Figure 14 As shown in S1031, if the power supply current of the electronic device exceeds a threshold, or if the temperature of the electronic device exceeds a threshold, the electronic device will trigger overheat protection. Therefore, the image processing module will not perform second-stage image processing on the image to reduce the risk of the electronic device crashing or restarting. Since the overheat protection triggering of the electronic device has the highest priority, the image processing module will not perform second-stage image processing on the image even if other conditions are met when the overheat protection triggering condition is met.
[0097] like Figure 14 As shown in S1032, if the electronic device does not trigger overheat protection, the camera application is open, and the electronic device does not enable split screen or small window, since the electronic device currently only displays the window of the camera application and the user's touch point is on the camera application, the image processing module does not perform second-stage image processing on the image, and prioritizes ensuring the camera application's foreground photo-taking task.
[0098] like Figure 14 As shown in S1033, if the camera application is open and the electronic device has enabled split-screen or small window mode, the electronic device displays not only the camera application window but also the windows of other applications. Furthermore, if the user's touch point is on the camera application, the image processing module does not perform second-stage image processing on the image, prioritizing the camera application's foreground photo-taking task.
[0099] like Figure 14 As shown in S1034, if the electronic device does not trigger overheat protection, the camera application is open, the electronic device is in split-screen or small window mode, the user's touch point is not on the camera application, the user is not interacting with the image, and the electronic device is in power-saving mode, then the image processing module will not perform the second stage of image processing. Alternatively, if the electronic device does not trigger overheat protection, the camera application is not open, the user is not interacting with the image, and the electronic device is in power-saving mode, then the image processing module will not perform the second stage of image processing. This reduces the power consumption of the electronic device and extends its battery life.
[0100] like Figure 14 As shown in S1035, if the electronic device does not trigger overheat protection, the camera application is open, the electronic device is in split-screen or small window mode, the user's touch point is not on the camera application, the user is not interacting with the image, the electronic device is not in power-saving mode, or the electronic device's system resources are insufficient, then the image processing module will not perform the second-stage image processing. Alternatively, if the electronic device does not trigger overheat protection, the camera application is not open, the user is not interacting with the image, the electronic device is not in power-saving mode, or the electronic device's system resources are insufficient, then the image processing module will not perform the second-stage image processing. This ensures the smooth operation of other foreground applications, achieving the goal of staggered operation of the second-stage image processing and other high-load applications, thus improving the smoothness of electronic device use.
[0101] It should be noted that in S1034 and S1035, the order of the conditions—whether the electronic device is in power-saving mode and whether system resources are sufficient—can be interchanged. For example:
[0102] like Figure 15As shown in S1038, if the electronic device does not trigger overheat protection, the camera application is open, the electronic device has split-screen or a small window enabled, the user's touch point is not on the camera application, the user is not interacting with the image, and the electronic device's system resources are insufficient, then the image processing module will not perform the second-stage image processing. Alternatively, if the electronic device does not trigger overheat protection, the camera application is not open, the user is not interacting with the image, and the electronic device's system resources are insufficient, then the image processing module will not perform the second-stage image processing. This ensures the smooth operation of other foreground applications, achieving the goal of staggering the second-stage image processing with other high-load applications, and improving the smoothness of electronic device use.
[0103] like Figure 15 As shown in S1039, if the electronic device does not trigger overheat protection, the camera application is open, the electronic device is in split-screen or small window mode, the user's touch point is not on the camera application, the user is not interacting with the image, the electronic device has sufficient system resources, and the electronic device is in power-saving mode, then the image processing module will not perform the second stage of image processing. Alternatively, if the electronic device does not trigger overheat protection, the camera application is not open, the user is not interacting with the image, the electronic device has sufficient system resources, and the electronic device is in power-saving mode, then the image processing module will not perform the second stage of image processing. This reduces the power consumption of the electronic device and extends its battery life.
[0104] like Figure 14 As shown in S1036, if the electronic device does not trigger overheat protection, the camera application is open, the electronic device is in split-screen or small window mode, the user's touch point is not on the camera application, the user is not interacting with the image, the electronic device is not in power-saving mode, and the electronic device has sufficient system resources, then the image processing module reads the original image information and performs second-stage image processing on the image based on the original image information. Alternatively, if the electronic device does not trigger overheat protection, the camera application is not open, the user is not interacting with the image, the electronic device is not in power-saving mode, and the electronic device has sufficient system resources, then the image processing module reads the original image information and performs second-stage image processing on the image based on the original image information. This fully utilizes the idle system resources of the electronic device to perform second-stage image processing on the image.
[0105] like Figure 14As shown in S1037, if the electronic device does not trigger overheat protection, the camera application is open, the electronic device has a split-screen or small window enabled, and the user's touch point is not on the camera application, but the user is manipulating the image, the image processing module reads the original image information and performs second-stage image processing based on the original image information. Alternatively, if the electronic device does not trigger overheat protection, the camera application is not open, and the user is manipulating the image, the image processing module reads the original image information and performs second-stage image processing based on the original image information. This fully utilizes the idle system resources of the electronic device, allowing for rapid second-stage image processing by interpolating when the user is manipulating the image, ensuring the user obtains the target image in the shortest possible time.
[0106] The image processing module performs a second stage of image processing on the original image information. This involves processing the original image data using image processing algorithms based on the camera's hardware parameters, software parameters, processor parameters, and environmental parameters. For example, the electronic device can be configured according to the camera's hardware parameters and then perform preprocessing operations on the original image data to obtain a preprocessed image. For instance, the ISP can perform preprocessing such as de-mosaicing, white balance adjustment, and noise reduction on the original image data based on the camera's software parameters. The electronic device then further processes the preprocessed image, such as sharpening, increasing contrast, and color correction. The electronic device can also further process the preprocessed image using image processing algorithms; for example, the NPU can perform scene recognition, cropping, and rotation of the image to finally obtain the target image. Compared to the thumbnail obtained when the camera is captured, this target image has better image quality.
[0107] It should be noted that if images have not undergone second-stage image processing for a certain period of time, these images will be processed first based on their original image information to avoid image loss. Secondly, if multiple images use the same image processing algorithm, they will be processed in batches based on their original image information to reduce the overhead of loading and initializing the same algorithm. Thirdly, because users are more likely to operate on recently captured images, for images of the same priority, a last-in, first-out (LIFO) principle will be followed, with the image processing module prioritizing the second-stage processing of the most recently captured image based on its original image information.
[0108] The photographing method and electronic device provided in this application perform a first-stage image processing to obtain a thumbnail when taking a picture. The method determines whether to perform a second-stage image processing based on the original image information based on the following information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or a small window, whether the touch point is in the camera application, whether the user is interacting with the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient. This allows the first and second stages of image processing to run independently, and more strictly limits the triggering conditions for the second stage of image processing, preventing it from affecting the smoothness of other foreground applications.
[0109] like Figure 16 As shown, this application embodiment also provides a chip system. The chip system 160 includes at least one processor 1601 and at least one interface circuit 1602. The at least one processor 1601 and at least one interface circuit 1602 are interconnected via lines. The processor 1601 is used to support electronic devices in implementing the various steps in the above method embodiments, for example... Figure 8 , Figure 14 , Figure 15 The method shown allows at least one interface circuit 1602 to be used to receive signals from other devices (e.g., memory) or to send signals to other devices (e.g., a communication interface). The chip system may include a chip and may also include other discrete components.
[0110] This application also provides a computer-readable storage medium including instructions that, when executed on the electronic device, cause the electronic device to perform the steps described in the method embodiments, such as executing... Figure 8 , Figure 14 , Figure 15 The method shown.
[0111] This application also provides a computer program product including instructions, which, when executed on the aforementioned electronic device, cause the electronic device to perform the various steps in the method embodiments described above, such as executing... Figure 8 , Figure 14 , Figure 15 The method shown.
[0112] The technical effects of the chip system, computer-readable storage medium, and computer program product are described in the preceding method embodiments.
[0113] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0114] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0115] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.
[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for taking pictures, applied to electronic devices, characterized in that, The method includes: Launch the camera app to display images captured by the camera in real time; In response to the user's shooting operation, the system acquires and stores the raw image information of the image captured by the camera, and performs a first-stage image processing on the image to obtain a thumbnail; the raw image information includes at least one of the following: raw image data, image processing algorithm involved in the second-stage image processing, camera hardware parameters, camera software parameters, processor parameters, and environmental parameters; In response to at least one of the following information, a second stage of image processing is performed on the image based on the original image information: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power saving mode, and whether the system resources are sufficient; if multiple images use the same image processing algorithm, the second stage of image processing is performed on the multiple images in batches based on the original image information of the multiple images.
2. The method according to claim 1, characterized in that, The second-stage image processing is performed on the image based on the original image information in response to at least one of the following: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient, including: If the electronic device does not trigger overheat protection, the camera application is open, the electronic device is in split-screen or small window mode, the user's touch point is not on the camera application, the user does not interact with the image, the electronic device is not in power-saving mode, and the electronic device has sufficient system resources, then the original image information is read and the image is processed in the second stage based on the original image information.
3. The method according to claim 1, characterized in that, The second-stage image processing is performed on the image based on the original image information in response to at least one of the following: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient, including: If the electronic device does not trigger overheat protection, the camera application is not open, the user does not interact with the image, the electronic device is not in power-saving mode, and the electronic device has sufficient system resources, then the original image information is read, and the image is processed in the second stage based on the original image information.
4. The method according to claim 1, characterized in that, The second-stage image processing is performed on the image based on the original image information in response to at least one of the following: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient, including: If the electronic device does not trigger overheat protection, the camera application is open, the electronic device has split-screen or small window enabled, the user's touch point is not on the camera application, and the user operates on the image, then the original image information of the image is read, and the image is processed in the second stage based on the original image information.
5. The method according to claim 1, characterized in that, The second-stage image processing is performed on the image based on the original image information in response to at least one of the following: whether the electronic device has triggered overheat protection, whether the camera application is open, whether the electronic device has enabled split-screen or small window, whether the touch point is in the camera application, whether the user is operating the image, whether the electronic device is in power-saving mode, and whether system resources are sufficient, including: If the electronic device does not trigger overheat protection, the camera application is not open, and the user operates on the image, the original image information is read, and the image is processed in the second stage based on the original image information.
6. The method according to any one of claims 1-5, characterized in that, Also includes: If there are images that have not undergone the second stage of image processing for a certain period of time, the second stage of image processing will be performed on these images based on their original image information.
7. An electronic device, characterized in that, The device includes a camera, a display screen, a processor, and a memory. The processor controls the display screen to display images and controls the camera to capture images. The memory stores instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed on the electronic device, cause the electronic device to perform the method as described in any one of claims 1-6.
9. A computer program product, characterized in that, Includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-6.
Citation Information
Patent Citations
Image processing method and electronic equipment
CN116055859A
