Image processing method and related equipment

Abstract

The present application provides an image processing method and related equipment, which relates to the field of image processing, and the method includes: determining the memory usage for the x-th number of photos; when it is less than the first memory threshold, determining the x-level processing algorithm as the first photo algorithm, and determining the x-level processing frame number as the first frame number; when it is greater than or equal to the first memory threshold, but less than the second memory threshold, determining the x-level processing algorithm as the second photo algorithm, and determining the x-level processing frame number as the second frame number; when it is greater than the second memory threshold, determining the x-level processing algorithm as the third photo algorithm, and determining the x-level processing frame number as the third frame number; using the x-level processing algorithm to process the x-level processing frames in the original image to obtain the corresponding captured image; saving the captured image. The present application can reduce the pressure of memory increment and realize continuous and rapid photo taking by distinguishing the memory usage and adaptively selecting processing algorithms of different durations and / or different processing frame numbers.

Description

[0001] This application is a divisional application of the Chinese patent application submitted to the State Intellectual Property Office on August 25, 2022, with application number 202211026658.0 and application name "Photographing method and related equipment". Technical Field

[0002] The present application relates to the field of image technology, and in particular, to an image processing method and related equipment. Background Art

[0003] With the popularity of electronic devices with camera functions in life, taking photos with electronic devices has become a daily behavior.

[0004] At present, most electronic devices use multi-frame photo algorithms to achieve better photo quality when processing photos. However, the more frames the algorithm processes, the longer the corresponding processing time. In this case, if the user triggers multiple photos in succession, a photo request will appear in the background. The accumulated images to be processed will cause the memory usage to continue to expand. When the remaining memory is not enough to support the photo processing, the background will no longer respond to the user's photo command, that is, the electronic device will not be able to continue taking photos. A new method is urgently needed to solve this problem. Summary of the invention

[0005] The present application provides an image processing method and related equipment, which can reduce the pressure of memory increment and achieve continuous and rapid photography by distinguishing the memory usage and adaptively selecting processing algorithms with different processing times and / or processing frame numbers.

[0006] In a first aspect, a photographing method is provided, which is applied to an electronic device including a camera, and the method includes:

[0007] Display a first interface, wherein the first interface includes a first control;

[0008] When a first operation on the first control is detected, in response to the first operation, an original image is captured once;

[0009] Determine the memory usage for the xth number of photos;

[0010] When the memory usage is less than the first memory threshold, determining the x-level processing algorithm as the first photographing algorithm, and determining the x-level processing frame number as the first frame number, where x is an integer greater than 1;

[0011] When the memory occupancy is greater than or equal to the first memory threshold but less than the second memory threshold, determining the x-level processing algorithm to be the second photographing algorithm, and determining the x-level processing frame number to be the second frame number;

[0012] When the memory usage is greater than the second memory threshold, determining that the x-level processing algorithm is a third photographing algorithm, and determining that the x-level processing frame number is a third frame number, and the first memory threshold is less than the second memory threshold;

[0013] Using the x-level processing algorithm to process the x-level processing frames in the original image to obtain a corresponding captured image;

[0014] Saving the captured image;

[0015] Wherein, when the first photographing algorithm, the second photographing algorithm and the third photographing algorithm are the same, the first frame number, the second frame number and the third frame number gradually decrease;

[0016] When the processing times of the first photographing algorithm, the second photographing algorithm, and the third photographing algorithm gradually decrease, the first frame number, the second frame number, and the third frame number are the same or gradually decrease.

[0017] Optionally, the first operation is a click operation. The first operation may also include a voice instruction operation or other operations for instructing the electronic device to take a photo.

[0018] An embodiment of the present application provides a photographing method. In response to a first operation of a user on a first control, the electronic device uses a camera to collect an original image once. Then, by determining the size of the memory usage and adaptively selecting algorithms with different processing times for processing according to the size of the memory usage, for example, when the memory usage is small, an algorithm with a longer processing time is selected for processing, and when the memory usage gradually increases, an algorithm with a shorter processing time is selected for processing. This can reduce the pressure of memory increment in the electronic device, improve the efficiency of photographing processing, respond to the user's photographing needs in a timely manner, and achieve continuous and rapid photographing.

[0019] In a possible implementation manner of the first aspect, the method further includes:

[0020] For the x-th photo-taking number, when the device temperature is greater than or equal to the temperature threshold, one frame of the original image is processed using a single-frame algorithm to obtain a corresponding photographed image;

[0021] Wherein, when the first photographing algorithm, the second photographing algorithm and the third photographing algorithm are the same, the processing time of the single-frame algorithm is shorter than the processing time of the third photographing algorithm;

[0022] When the processing times of the first photographing algorithm, the second photographing algorithm and the third photographing algorithm gradually decrease, the processing time of the single-frame algorithm is the same as the processing time of the third photographing algorithm, or the processing time of the single-frame algorithm is shorter than the processing time of the third photographing algorithm.

[0023] The device temperature refers to the temperature inside the electronic device.

[0024] In this implementation, when the device temperature is higher than the temperature threshold, it can switch to a single-frame algorithm to process one frame of the original image, thereby minimizing the workload of the electronic device, slowing down the increase in heat, and reducing the incremental heat pressure, so that the electronic device can take more photos before it stops working, thereby meeting the user's photo needs.

[0025] It should be understood that since the single-frame algorithm can be processed only in the image signal processor, the processing speed of the single-frame algorithm is relatively fast and the processing time is relatively short.

[0026] In a possible implementation manner of the first aspect, the method further includes:

[0027] When the first operation corresponds to the first photo-taking number, performing environmental detection for the scene to be photographed;

[0028] Determine the number of level 1 processing frames and the level 1 processing algorithm corresponding to the first number of photographs according to the environmental detection result;

[0029] The first-level processing algorithm is used to process the first-level processing frame number in the original image collected corresponding to the first shooting number to obtain the corresponding captured image.

[0030] In this implementation, the present application can determine different scenes by performing environmental detection on the scene to be shot, and then select different algorithms based on the different subdivided scenes, and select different numbers of frames from the corresponding collected original images for processing, so as to adaptively improve the quality and effect of the captured images corresponding to the first number of photos in each scene.

[0031] In a possible implementation of the first aspect, the environment detection includes at least illumination detection and dynamic range detection, and the environment detection result includes at least illumination and dynamic range values;

[0032] Determining the number of level 1 processing frames and the level 1 processing algorithm corresponding to the first number of photographing times according to the environmental detection result includes:

[0033] When the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, determining the number of level 1 processing frames to be a1 frames, and determining the level 1 processing algorithm to be the A1 algorithm;

[0034] When the illumination is less than the illumination threshold and the dynamic range value is less than the dynamic range threshold, determining the number of level 1 processing frames to be b1 frames, and determining the level 1 processing algorithm to be the B1 algorithm;

[0035] When the illumination is greater than or equal to the illumination threshold, and the dynamic range value is greater than or equal to the dynamic range threshold, determining the number of level 1 processing frames to be c1 frames, and determining the level 1 processing algorithm to be the C1 algorithm;

[0036] When the illumination is greater than or equal to the illumination threshold and the dynamic range value is less than the dynamic range threshold, determining the number of level 1 processing frames to be d1 frames, and determining the level 1 processing algorithm to be the D1 algorithm;

[0037] Wherein, a1, b1, c1 and d1 are all integers greater than or equal to 1.

[0038] Optionally, the a1 frame, b1 frame, c1 frame or d1 frame may be selected from all frames of the original image in the acquisition order, or the a1 frame, b1 frame, c1 frame or d1 frame may be arbitrarily extracted from all frames of the original image.

[0039] In a possible implementation of the first aspect, using the level 1 processing algorithm, processing the level 1 processing frame number in the original image collected corresponding to the first shooting number to obtain the corresponding captured image includes:

[0040] When the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, the A1 algorithm is used to process the a1 frame in the original image collected corresponding to the first shooting time to obtain a corresponding captured image;

[0041] When the illumination is less than the illumination threshold and the dynamic range value is less than the dynamic range threshold, the b1 frame in the original image collected corresponding to the first shooting time is processed by the B1 algorithm to obtain a corresponding captured image;

[0042] When the illumination is greater than or equal to the illumination threshold, and the dynamic range value is greater than or equal to the dynamic range threshold, the C1 algorithm is used to process the C1 frame in the original image collected corresponding to the first shooting time to obtain a corresponding captured image;

[0043] When the illumination is greater than or equal to the illumination threshold and the dynamic range value is less than the dynamic range threshold, the D1 algorithm is used to process the d1 frame in the original image collected corresponding to the first shooting time to obtain a corresponding captured image.

[0044] Optionally, when the dynamic range value is greater than or equal to the dynamic range threshold, the first photo-taking number corresponds to the a1 frame in the captured original image, and / or the first photo-taking number corresponds to the c1 frame in the captured original image, including at least one of a long-exposure original image, a normal-exposure original image, and a short-exposure original image.

[0045] In this implementation, when the processed original images include original images with long exposure, normal exposure and short exposure, the dynamic range and image details of the image can be adjusted so that the content presented by the obtained captured image is more realistic and the image quality is relatively better.

[0046] In a possible implementation manner of the first aspect, after determining that the x-level processing algorithm is a third photographing algorithm and determining that the x-level processing frame number is a third frame number, the method further includes:

[0047] Determine a frequency of a first operation on the first control, and a difference between a memory watermark and the memory usage;

[0048] When the frequency meets the preset frequency condition or the difference is less than the preset difference, determining the number of photos to be taken in the background waiting;

[0049] The x-level processing algorithms corresponding to the number of times of photos waiting in the background are respectively used as initial processing algorithms;

[0050] The x-level processing algorithm corresponding to the most recently determined number of photographs is used as the target processing algorithm;

[0051] Using the target processing algorithm to replace the initial processing algorithm of each of the number of photos waiting in the background;

[0052] When any one of the photographing times waiting in the background is called for processing, the target processing algorithm is used to process the x-level processing frames in the original image corresponding to the any one of the photographing times to obtain a corresponding photographed image.

[0053] In this implementation, since the processing algorithm corresponding to the most recently determined number of photographs is determined based on the current memory situation, the processing algorithm for the waiting number of photographs is adjusted accordingly, which can control the degree of memory accumulation, reduce memory usage, and achieve continuous and rapid photographing.

[0054] In a possible implementation manner of the first aspect, the first interface refers to a photographing interface, and the first control refers to a control for instructing photographing.

[0055] In a possible implementation manner of the first aspect, the first interface refers to a video recording interface, and the first control refers to a control for instructing snapshot.

[0056] In a second aspect, an image processing apparatus is provided, the apparatus comprising a unit for executing each step in the above first aspect or any possible implementation manner of the first aspect.

[0057] In a third aspect, an electronic device is provided, comprising one or more processors and a memory;

[0058] The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method provided in the above first aspect or any possible implementation of the first aspect.

[0059] In a fourth aspect, a chip is provided, which is applied to an electronic device, and the chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0060] In a fifth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by an electronic device, the electronic device executes the method provided in the first aspect or any possible implementation of the first aspect.

[0061] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program code, when the computer program code is executed by an electronic device, the electronic device executes the method provided in the first aspect or any possible implementation of the first aspect.

[0062] The embodiment of the present application provides an image processing method and related devices, in response to a first operation of a user on a first control, the electronic device uses a camera to collect an original image. When the first operation corresponds to the first photo-taking number, an environmental detection is performed for the scene to be photographed, and then the first-level processing frame number and the first-level processing algorithm corresponding to the first photo-taking number are determined according to the environmental detection result. When the first operation corresponds to the second photo-taking number and subsequent other photo-taking numbers, the memory usage is first detected, the determined memory usage size is compared with the preset memory threshold, and then different processing frames and processing algorithms are selected according to different memory usage sizes.

[0063] When processing the first number of photos, since the scene is subdivided according to the environmental detection results, different algorithms and frame rates are selected according to different scenes, so that the quality and effect of the captured images corresponding to the first number of photos in each scene can be adaptively improved.

[0064] When processing the second photo and subsequent photos, since the memory usage is subdivided, different algorithms and frame rates are selected according to different memory usage. When the memory usage is larger, the algorithm with shorter processing time and / or the number of processing frames is reduced, thereby reducing the pressure of memory increment, improving processing efficiency, and meeting the needs of fast photo taking. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of a scenario applicable to the embodiments of the present application;

[0066] Figure 2 This is another schematic diagram of a scenario to which the embodiment of the present application is applicable;

[0067] Figure 3 It is a flowchart of a photographing method provided in an embodiment of the present application;

[0068] Figure 4 This is a schematic diagram of a click operation interface provided by an embodiment of the present application;

[0069] Figure 5 This is another interface schematic diagram of a click operation provided in an embodiment of the present application;

[0070] Figure 6 is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0071] Figure 7 It is a schematic diagram of a process of environmental detection provided by an embodiment of the present application;

[0072] Figure 8 It is a table provided in the embodiment of the present application;

[0073] Fig. 9 It is a flowchart of determining the corresponding number of frames and an algorithm according to the memory usage provided by an embodiment of the present application;

[0074] Fig.10 This is a schematic diagram of the growth of memory usage provided by an embodiment of the present application;

[0075] Fig.11 is a flowchart of another photographing method provided in an embodiment of the present application;

[0076] Fig.12 It is a flowchart of another photographing method provided in an embodiment of the present application;

[0077] Fig.13 It is a flowchart of another photographing method provided in an embodiment of the present application;

[0078] Fig.14 is another form provided in the embodiment of the present application;

[0079] Fig.15 It is another table provided in the embodiment of the present application;

[0080] Fig.16 It is another table provided in the embodiment of the present application;

[0081] Fig.17 yes Fig.16 Schematic diagram of the corresponding processing process;

[0082] Fig.18 is a schematic diagram of a display interface of an electronic device provided in an embodiment of the present application;

[0083] Fig.19 is a schematic diagram of a display interface of another electronic device provided in an embodiment of the present application;

[0084] Fig. 20 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0085] Fig.21 It is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0086] Fig. 22 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0087] Fig.23 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0089] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0090] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0091] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0092] 1. RGB (red, green, blue) color space refers to a color model related to the structure of the human visual system. According to the structure of the human eye, all colors are regarded as different combinations of red, green and blue.

[0093] 2. YUV color space refers to a color encoding method, where Y represents brightness, and U and V represent chromaticity. The above RGB color space focuses on the human eye's perception of color, while the YUV color space focuses on the visual sensitivity to brightness. The RGB color space and the YUV color space can be converted to each other.

[0094] 3. Pixel value refers to a set of color components corresponding to each pixel in a color image in the RGB color space. For example, each pixel corresponds to a set of three primary color components, where the three primary color components are red component R, green component G, and blue component B.

[0095] 4. Bayer pattern color filter array (CFA). When an image is converted from an actual scene into image data, the image sensor usually receives the red channel signal, the green channel signal, and the blue channel signal respectively, and then synthesizes the information of the three channel signals into a color image. However, in this solution, three filters are required at each pixel position, which is expensive and difficult to make. Therefore, a layer of color filter array can be covered on the surface of the image sensor to obtain the information of the three channel signals. The Bayer pattern color filter array refers to the arrangement of filters in a checkerboard format. For example, the minimum repeating unit in the Bayer pattern color filter array is: a filter for obtaining a red channel signal, two filters for obtaining a green channel signal, and a filter for obtaining a blue channel signal arranged in a 2×2 manner.

[0096] 5. Bayer format image, that is, the image output by the image sensor based on the Bayer format color filter array. The pixels of multiple colors in the image are arranged in Bayer format. Among them, each pixel in the Bayer format image corresponds to only one color channel signal. Exemplarily, since human vision is more sensitive to green, it can be set that green pixels (pixels corresponding to green channel signals) account for 50% of all pixels, blue pixels (pixels corresponding to blue channel signals) and red pixels (pixels corresponding to red channel signals) each account for 25% of all pixels. Among them, the minimum repeating unit of the Bayer format image is: one red pixel, two green pixels and one blue pixel arranged in a 2×2 manner. The Bayer format image is an image located in the RAW domain.

[0097] 6. Shooting parameters

[0098] Shooting parameters may include shutter speed, exposure time, aperture value (AV), exposure value (EV) and ISO, which are introduced below.

[0099] The shutter is a device that controls the length of time light enters the camera to determine the exposure time of the image. The longer the shutter remains open, the more light enters the camera and the longer the corresponding exposure time of the image. Conversely, the shorter the shutter remains open, the less light enters the camera and the shorter the corresponding exposure time of the image.

[0100] Exposure time refers to the time the shutter is open in order to project light onto the photosensitive surface of the camera's photosensitive material. The exposure time is determined by the sensitivity of the photosensitive material and the illumination on the photosensitive surface. The longer the exposure time, the more light enters the camera, and the shorter the exposure time, the less light enters the camera. Therefore, a long exposure time is required in a dark scene, and a short exposure time is required in a backlit scene.

[0101] The aperture value (f-number) is the ratio of the focal length of the lens in the camera to the diameter of the lens. The larger the aperture value, the more light enters the camera. The smaller the aperture value, the less light enters the camera.

[0102] The exposure value is a combination of exposure time and aperture value to represent the light transmission capacity of the camera lens. The exposure value can be defined as:

[0103]

[0104] Where N is the aperture value; t is the exposure time in seconds.

[0105] ISO is used to measure the sensitivity of film to light, i.e., sensitivity or gain. For insensitive film, a longer exposure time is required to achieve an image with the same brightness as sensitive film. For sensitive film, a shorter exposure time is required to achieve an image with the same brightness as insensitive film.

[0106] Among the shooting parameters, shutter, exposure time, aperture value, exposure value and ISO, the electronic device can implement at least one of auto focus (AF), automatic exposure (AE), and auto white balance (AWB) through an algorithm to achieve automatic adjustment of these shooting parameters.

[0107] Autofocus means that the electronic device adjusts the position of the focusing lens to obtain the highest image frequency component to obtain a higher image contrast. Among them, focusing is a continuous accumulation process. The electronic device compares the contrast of images taken by the lens in different positions to obtain the position of the lens when the image contrast is the largest, and then determines the focal length for focusing.

[0108] Automatic exposure refers to the electronic device automatically setting the exposure value according to the available light conditions. The electronic device can automatically set the shutter speed and aperture value according to the exposure value of the current captured image to achieve automatic exposure setting.

[0109] The color of an object will change due to the color of the projected light, and the images captured by electronic devices will have different color temperatures under different light colors. White balance is closely related to the surrounding light. Regardless of the ambient light, the camera of an electronic device can recognize white and restore other colors based on white. Automatic white balance allows electronic devices to adjust the fidelity of image colors according to light source conditions. 3A refers to auto focus, auto exposure, and auto white balance.

[0110] Exemplarily, the exposure value may be any one of -24, -4, -3, -2, -1, 0, 1, 2, 3, 4, and 24.

[0111] The exposure image corresponding to EV0 is used to indicate that when the electronic device implements exposure through an algorithm, the exposure image is captured by the determined exposure value 0. The exposure image corresponding to EV-2 is used to indicate that when the electronic device implements exposure through an algorithm, the exposure image is captured by the determined exposure value -2. The exposure image corresponding to EV1 is used to indicate that when the electronic device implements exposure through an algorithm, the exposure image is captured by the determined exposure value 1. The rest is by analogy and will not be repeated here.

[0112] Among them, every increase of 1 in the exposure value will change the exposure by one level, that is, the exposure amount (referring to the integral of the light intensity received by a certain surface element on the surface of the object within the time t) is doubled, such as doubling the exposure time or the aperture area. Then, the increase in the exposure value will correspond to a slower shutter speed and a smaller f-value. It can be seen that EV0 has an increase of 2 in the exposure value relative to EV-2, which changes the exposure by two levels; similarly, EV1 has an increase of 1 in the exposure value relative to EV0, which changes the exposure by one level.

[0113] Here, when the exposure value EV is equal to 0, the exposure value is generally the best exposure value under the current lighting conditions. Accordingly, the exposure image correspondingly acquired by the electronic device under the condition of EV0 is the best exposure image under the current lighting conditions, and the best exposure image can also be called a reference exposure image.

[0114] It should be understood that the “optimal” exposure image refers to an exposure image determined by a given electronic device through an algorithm, and when the electronic device is different, the algorithm is different, or the current lighting conditions are different, the determined optimal exposure image will be different.

[0115] The above is a brief introduction to the terms involved in the embodiments of the present application, which will not be repeated below.

[0116] With the popularity of electronic devices with camera functions in life, taking photos with electronic devices has become a daily behavior.

[0117] Most electronic devices use multi-frame photo algorithms to achieve better photo quality when processing photos. However, the more frames the algorithm processes, the longer the corresponding processing time. For example, when the number of processed images is 9 frames, the time from the user pressing the shooting button displayed on the display interface of the electronic device to the time when the image processing is completed and available for the user to view, that is, the end-to-end duration of the algorithm is about 2.5s; when the number of processed images is greater than 9 frames, the end-to-end duration of the algorithm will be greater than 2.5s.

[0118] When an electronic device performs photo processing, it will only perform the next photo processing after the previous photo processing is completed. Based on this single-threaded processing, if the user triggers multiple photo commands in succession, and the interval between triggering the photo is less than the above 2.5s, the electronic device will not be able to respond in time because the photo algorithm takes a long time to process each photo, resulting in an increasing number of photos waiting in the background. As the number of waiting photos increases, the accumulated data to be processed in the background also increases, which continuously expands the memory usage. When the remaining memory is insufficient to support the photo processing, the background will no longer respond to the photo command triggered by the user.

[0119] In this way, no matter how the user clicks the camera button, it feels like the camera button cannot be pressed. At this time, the electronic device is unable to take pictures normally, resulting in a very poor experience for the user.

[0120] In view of this, an embodiment of the present application provides a photo-taking method. In response to a first operation of a user on a first control, the electronic device uses a camera to capture an original image once, and then determines the size of the memory usage and adaptively selects algorithms with different processing times for processing based on the size of the memory usage. For example, when the memory usage is small, an algorithm with a longer processing time is selected for processing, and when the memory usage gradually increases, an algorithm with a shorter processing time is selected for processing. This can reduce the pressure of memory increment in the electronic device, improve the efficiency of photo processing, respond to the user's photo needs in a timely manner, and achieve continuous and fast photo taking.

[0121] The photographing method provided in the embodiment of the present application can be applied to various electronic devices.

[0122] In some embodiments of the present application, the electronic device may be various camera devices such as sports cameras and digital cameras, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc., or may be other devices or apparatuses capable of displaying images. The embodiments of the present application do not impose any restrictions on the specific type of the electronic device.

[0123] The following description takes a mobile phone as an example of an electronic device. The mobile phone may include a display screen and one or more cameras. The mobile phone uses one or more cameras to shoot in a scene to be shot to obtain images with different characteristics. The display screen is used to display the shot image obtained by post-shooting processing.

[0124] The scene to be photographed refers to the scene that the user wants to photograph. If the user uses the camera of the electronic device to aim at a scene including an object, the scene including the object is the scene to be photographed. It should be understood that the scene to be photographed does not refer to a specific scene, but is a scene that is aimed at in real time following the direction of the camera.

[0125] Based on this, combined Figure 1 and Figure 2 The application scenarios provided by the embodiments of the present application are introduced. Figure 1 A schematic diagram of a scenario applicable to the embodiment of the present application is shown. Figure 2 Another schematic diagram of a scenario to which the embodiment of the present application is applicable is shown.

[0126] Application scenario 1: Photo shooting

[0127] like Figure 1 As shown in (a) of FIG. 1 , a camera application is installed on the electronic device 100. In addition, multiple applications are installed, and the present application embodiment does not impose any restrictions on this. Exemplarily, in response to a user's click operation on the camera application, when the electronic device 100 runs the camera application, the electronic device 100 displays the following Figure 1 The shooting interface shown in (b).

[0128] The shooting interface includes multiple shooting modes of the camera application, such as large aperture mode 41, night scene mode 42, portrait mode 43, photo mode 44, video mode 45, etc. The shooting interface also includes a first control, which is a shooting key 50. The shooting key 50 is used to indicate the current shooting mode. For example, when the camera is turned on, the shooting key 50 indicates by default that the current shooting mode is the photo mode 44.

[0129] like Figure 1 As shown in (b) of FIG. 1 , the shooting interface also includes a viewfinder window 60, which can be used to display a preview image before shooting in real time. The shooting interface also displays a second control, which is a zoom option 61. The user can select the currently required zoom factor in the zoom option 61, for example, 0.5 times, 2 times or 50 times zoom factor, etc.

[0130] Combination Figure 1 As shown in (a) and (b), when the user turns on the camera and the current shooting mode is the default photo mode, after the user points the camera on the mobile phone at the scene to be photographed, the viewfinder 60 can display the preview image before the photo is taken in real time; then, in response to the user's continuous multiple clicks on the shooting key 50, the electronic device can call the photo method provided by the present application, and adaptively select different photo algorithms for processing according to the size of the memory, thereby reducing the pressure of memory increment, improving the efficiency of photo processing, responding to the user's photo needs in a timely manner, realizing fast photo taking, and avoiding the situation in the related technology where the background cannot respond to the photo command.

[0131] Application scenario 2: Scenes captured in video

[0132] like Figure 2 As shown in (a) of FIG. 1 , a camera application is installed on the electronic device 100. Exemplarily, in response to a user's click operation on the camera application, when the electronic device 100 runs the camera application, the electronic device displays the following Figure 2 The shooting interface shown in (b).

[0133] Among them, the description of the shooting interface and Figure 1 When the camera is turned on, the default shooting key 50 indicates that the current shooting mode is the photo shooting mode 44. Exemplarily, in response to a left sliding operation by the user, the shooting mode can be switched from the photo shooting mode 44 to the video recording mode 45.

[0134] like Figure 2As shown in (c) of FIG. 1 , when the shooting mode is switched to the video recording mode 45, the viewfinder window 60 included in the shooting interface can be used to display a preview image before recording in real time. For example, in response to a user clicking operation on the shooting key 50, when the electronic device 100 starts recording, the electronic device can display the following Figure 2 The recording interface shown in (d).

[0135] like Figure 2 As shown in (d) in the figure, the video recording interface can display the currently captured video image (as shown in the figure, the man walking with a shopping bag), the shooting progress (as shown in the figure, 00:12), the status icon (as shown in the figure, the shooting icon in front of the shooting progress is used to indicate that the current shooting state is in progress), and a third control, for example, a snapshot control 70. Of course, the shooting interface can also include other controls, for example: a zoom option 61, an end control 80, a pause / resume control 90, etc., which is not limited in this embodiment of the present application.

[0136] The pause / resume control 90 is used to display a pause icon during video shooting (see Figure 2 ), and is also used to pause the current video shooting process when the pause icon is clicked; and to display the shooting icon when the video shooting is paused, and to continue the current video shooting process when the user clicks the shooting icon ( Figure 2 The end control 80 is used to end the current video shooting process. The snapshot control 70 is used to capture a photo without pausing the current video shooting process and without ending the current video shooting process.

[0137] Combination Figure 2 As shown in (c) and (d), when the user wants to capture a photo of a man walking with a shopping bag, the user can click the capture control 70 in the shooting interface to obtain the captured photo. The captured photo is stored as an image in the gallery of the electronic device. If the user clicks the capture control 70 in the recording interface for multiple times in a row, the electronic device can call the photo taking method provided by the present application, and adaptively select different photo taking algorithms for processing according to the size of the memory, thereby reducing the pressure of memory increment, improving the efficiency of the capture processing, and promptly responding to the user's photo taking needs, achieving fast capture, and avoiding the situation in the related art where the background cannot respond to the photo taking command.

[0138] The above are two examples of application scenarios and do not limit the application scenarios of the present application.

[0139] Combine the following Figures 3 to 19 The photographing method provided in the embodiment of the present application is described in detail.

[0140] Figure 31 is a flow chart of a photographing method provided by the present application. The photographing method is applied to an electronic device including a camera, such as a mobile phone. Figure 3 As shown, the steps of the method may include the following S11 to S19.

[0141] S11. The electronic device displays a first interface, where the first interface includes a first control.

[0142] Alternatively, if Figure 1 As shown in (b), the first interface may be a shooting interface, and the first control is used to indicate a shooting key 50 in the shooting interface.

[0143] Alternatively, if Figure 2 As shown in (d) in FIG. 1 , the first interface may be a video recording interface, and the first control is used to indicate a snapshot control 70 in the video recording interface.

[0144] Of course, the first interface may also be other interfaces, and the first control may correspondingly be a control on other interfaces for instructing taking photos, and the embodiments of the present application do not impose any restrictions on this.

[0145] S12: When a first operation of the user on the first control is detected, in response to the first operation, the electronic device uses a camera to collect an original image.

[0146] Optionally, the first operation may be a click operation on the first control, or may be a voice instruction operation or other operation of instructing the electronic device to take a photo, and the embodiment of the present application does not impose any limitation on this.

[0147] The click operation refers to the behavior of a user touching the first control for a short time and then leaving the touchscreen.

[0148] For example, Figure 4 A schematic diagram of a click operation interface is shown, such as Figure 4 As shown in (a) and (b) in the figure, a click operation is recorded as the user's finger pressing the capture key 50 and then lifting it up. The time from the finger pressing the capture key 50 to the finger lifting the capture key 50 is the duration of a click operation, which is usually very short.

[0149] For example, Figure 5 Another interface diagram of a click operation is shown, such as Figure 5 As shown in (a) and (b), the user's finger pressing the snapshot control 70 and then lifting it is recorded as a click operation, and the time from the finger pressing the snapshot control 70 to the finger lifting the snapshot control 70 is the duration of one click operation.

[0150] Combination Figure 5For example, when the camera is turned on for recording, when the user clicks the capture control 70 once, in response to the click operation, the electronic device uses the camera to capture an original image once.

[0151] When the user performs multiple click operations on the snapshot control 70, in response to the multiple click operations, the electronic device can use the camera to collect multiple original images. It should be understood that when the user performs multiple click operations on the snapshot control 70, the first click operation corresponds to the first number of photos, the second click operation corresponds to the second number of photos, the third click operation corresponds to the third number of photos, and so on. No further description will be given for the subsequent operations.

[0152] Here, it should be noted that when the camera is turned off and then reopened, the number of photos needs to be counted again from 1. Alternatively, when the interval between two adjacent click operations exceeds the first preset interval and there is no data waiting to be processed in the background, the number of photos also needs to be counted again from 1. Among them, the first preset interval is at least greater than the processing time of one photo, and the specific duration of the first preset interval can be set and modified as needed, and the embodiment of the present application does not impose any restrictions on this.

[0153] For example, when the user opens the camera for the first time and clicks the capture button 50 for the first time, in response to the click operation, the electronic device uses the camera to capture an original image once, and the number of photos taken this time corresponds to the first number of photos taken. Then, the user closes and reopens the camera and clicks the capture button 50 again. In response to the current click operation, the electronic device uses the camera to capture an original image again. At this time, since the camera is reopened, the number of photos taken this time should be re-recorded as the first number of photos taken.

[0154] For example, Figure 6 A schematic diagram of the image processing process is provided.

[0155] like Figure 6As shown, when the user opens the camera application and performs the first click operation K1 on the shooting key 50, in response to the click operation K1, the electronic device uses the camera to collect an original image once, and the number of photos taken this time can be recorded as the first number of photos taken. However, after a long period of time, the user performs the next click operation K1' on the shooting key 50, and in response to the click operation K1', the electronic device uses the camera to collect the original image again. Since the interval between the click operation K1 and the click operation K1' exceeds the first preset interval time, and there is no other data waiting to be processed in the background, the corresponding number of photos needs to be accumulated from 1 again, that is, the number of photos taken this time is recorded as the first number of photos taken again; when the user performs the click operation K2' on the shooting key later, in response to the click operation K2', the electronic device uses the camera to collect the original image again, and since the interval between the click operation K2' and the click operation K1' is less than the first preset interval time, the number of photos taken this time can be recorded as the second number of photos taken. When the user performs a click operation K3' on the capture key 50 again, in response to the click operation K3', the electronic device uses the camera to collect the original image again. Since the interval time between the click operation K3' and the click operation K2' is less than the first preset interval time, the number of photos taken this time can be recorded as the third time of taking photos; and the same goes for the subsequent operations, which will not be repeated here.

[0156] It should be noted that if the interval between two adjacent click operations exceeds the first preset interval, but there is still data waiting to be processed in the background, it means that the user has performed many click operations before, and some data has not been processed in time due to the high click frequency and the slow background processing. At this time, the number of photos cannot be accumulated again from 1, and the original number of records should be maintained.

[0157] When the electronic device includes multiple cameras, in response to a click operation, one or more cameras may be used to capture the original image, and each camera may capture one or more frames of the original image. In other words, one or more frames of the original image may be captured each time, and the specific number of frames may be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0158] It should also be understood that the original image may be an image located in the RAW domain, but when the camera is a black and white camera, the original image may be a grayscale image; when the camera is a multispectral camera, the original image may be a multispectral image including multiple color channel signals. The format or characteristics of the original image changes with the camera, and the embodiments of the present application do not impose any restrictions on this.

[0159] In addition, the original image collected once may also include at least one of a long-exposure original image, a normal-exposure original image, and a short-exposure original image.

[0160] Among them, the long-exposure original image refers to the image obtained after a longer exposure time during shooting, and the short-exposure original image refers to the image obtained after a shorter exposure time during shooting. It should be understood that "long" and "short" are relative to the "normal" exposure time. When the exposure time corresponding to the normally exposed original image is different, the long exposure and short exposure will also change accordingly.

[0161] When multiple frames of original images include long-exposure original images, and / or normal-exposure original images, and / or short-exposure original images, the number of frames of the long-exposure original images, the number of frames of the normal-exposure original images, and the number of frames of the short-exposure original images can be selected and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0162] S13: When the first operation corresponds to the first photo-taking number, perform environmental detection on the scene to be photographed.

[0163] Among them, environmental detection at least includes illumination value (LV) detection and dynamic range detection. Of course, environmental detection can also include detection items, which can be set and modified according to needs. The embodiments of the present application do not impose any restrictions on this.

[0164] It should be understood that illumination detection refers to estimating the ambient brightness in the scene to be shot. Illumination refers to the degree to which an object is illuminated, specifically refers to the energy of visible light received by the object per unit area, referred to as illumination, and the unit is lux. Streets at night, dimly lit rooms, etc. are generally low illumination. In contrast, playgrounds under the sun, stages under spotlights, etc. can be called medium and high illumination. If the illumination of the scene to be shot is relatively low, when using an electronic device to shoot the scene to be shot, it will be blurry and the effect will be poor. If the illumination of the scene to be shot is relatively high, when using an electronic device to shoot the scene to be shot, it will be clearer and the shooting effect will be good.

[0165] Here, for example, a photoelectric sensor can be set in the electronic device to detect the illumination of the scene to be photographed. Alternatively, the illumination can be calculated by the following formula based on the exposure parameters of the camera, such as exposure time, sensitivity, aperture, etc., and the received response value:

[0166]

[0167] Among them, Exposure is the exposure time, Aperture is the aperture size, ISO is the sensitivity, and Luma is the average value of Y in the XYZ color space.

[0168] When the exposure parameters are the same, a higher response value indicates that the illumination of the scene to be photographed is also higher, and thus, the calculated illumination value is also larger.

[0169] It should be understood that dynamic range detection refers to the detection of the dynamic range value of the scene to be photographed. The dynamic range value is used to represent the brightness range of the pixels in the image, that is, the number of gray levels between the "brightest" pixel and the "darkest" pixel in the image. The larger the dynamic range value of an image, the richer the brightness levels it can represent, and the more realistic the visual effect of the image. The expression of the dynamic range value can be:

[0170]

[0171] Among them, dynamic range is the dynamic range value, bright is the brightness of the "brightest" pixel, and dark is the brightness of the "darkest" pixel. The unit of dynamic range is stop.

[0172] Here, according to the size of the detected dynamic range value, it can be determined whether the scene to be shot is a low dynamic range (LDR) scene or a high dynamic range (HDR) scene. Among them, the low dynamic range scene refers to a scene where the ambient light intensity is all low or all high, and the dynamic range is relatively narrow; the high dynamic range scene refers to a scene where the light intensity is low in some parts and high in other parts, and the dynamic range is relatively wide.

[0173] S14. Determine the number of level 1 processing frames and the level 1 processing algorithm corresponding to the first number of photographs according to the environmental detection result.

[0174] Combining the illumination detection and dynamic range detection methods, Figure 7 A schematic diagram of a process of environmental detection provided in an embodiment of the present application.

[0175] Among them, since the above S13 includes: when the first operation corresponds to the first photo shooting number, the environment detection is performed for the scene to be photographed, and the environment detection includes illumination detection, therefore, correspondingly, the environment detection result includes the detected illumination; similarly, since the environment detection also includes dynamic range detection, correspondingly, the environment detection result also includes the detected dynamic range value. Among them, the methods of illumination detection and dynamic range detection can refer to the above description, and will not be repeated here.

[0176] Combined with the illumination and dynamic range values ​​in the detected environmental test results, such as Figure 7 As shown, the above S14 may include the following S141 to S146.

[0177] S141. Determine whether the detected illuminance is less than an illuminance threshold.

[0178] Among them, the illumination threshold can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0179] S142: Determine whether the detected dynamic range value is less than a dynamic range threshold.

[0180] Among them, the dynamic range threshold can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0181] S143, such as Figure 8 As shown, when the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, or when the scene to be photographed is a low-brightness, high dynamic range scene, the number of level 1 processing frames corresponding to the first photo shooting time is determined to be a1 frames, and the A1 algorithm is determined to be the level 1 processing algorithm.

[0182] Wherein, a1 is all or part of the frames in the original image collected corresponding to the first shooting number.

[0183] S144, such as Figure 8 As shown, when the illumination is less than the illumination threshold and the dynamic range value is less than the dynamic range threshold, or when the scene to be photographed is a low-brightness, low-dynamic-range scene, the number of level 1 processing frames corresponding to the first photo-taking time is determined to be b1 frames, and the B1 algorithm is determined to be the level 1 processing algorithm.

[0184] Wherein, b1 is all or part of the frames in the original image collected corresponding to the first shooting number.

[0185] S145, such as Figure 8 As shown, when the illumination is greater than or equal to the illumination threshold and the dynamic range value is also greater than or equal to the dynamic range threshold, or when the scene to be photographed is a high-brightness, high-dynamic-range scene, the number of level 1 processing frames corresponding to the first photo-taking time is determined to be c1 frames, and the C1 algorithm is determined to be the level 1 processing algorithm.

[0186] Wherein, c1 is all or part of the frames in the original image collected corresponding to the first shooting number.

[0187] S146, such as Figure 8 As shown, when the illumination is greater than or equal to the illumination threshold and the dynamic range value is less than the dynamic range threshold, or when the scene to be photographed is a high-brightness, low-dynamic-range scene, the number of level 1 processing frames corresponding to the first photo-taking time is determined to be d1 frames, and the D1 algorithm is determined to be the level 1 processing algorithm.

[0188] Wherein, d1 is all or part of the frames in the original image collected corresponding to the first shooting number.

[0189] It should be understood that A1, B1, C1 and D1 are used to refer to the names of algorithms. A1 algorithm, B1 algorithm, C1 algorithm and D1 algorithm can indicate the same algorithm or different algorithms. They can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0190] At the same time, a1, b1, c1 and d1 are used to refer to the number of processed frames. a1, b1, c1 and d1 can indicate the same number of frames or different numbers of frames. The specific number of frames can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0191] Among them, the a1 frame, b1 frame, c1 frame or d1 frame can be selected from all the frames of the original image in the acquisition order for processing, or the a1 frame, b1 frame, c1 frame or d1 frame can be arbitrarily extracted from all the frames of the original image for processing, or the images captured by one or more cameras can be selected in a designated camera manner to be processed as the a1 frame, b1 frame, c1 frame or d1 frame. Of course, the method of selecting the a1 frame, b1 frame, c1 frame or d1 frame can be the same or different, and can be specifically set and modified as needed, and the embodiment of the present application does not impose any restrictions on this.

[0192] In addition, for a scene to be captured with a high dynamic range, the determined a1 frame or c1 frame may include at least one of a long-exposure original image, a normal-exposure original image, and a short-exposure original image, or at least one of a long-exposure original image, a normal-exposure original image, and a short-exposure original image may be additionally added. The specific settings and modifications may be made as needed, and the embodiments of the present application do not impose any restrictions on this.

[0193] It should be understood that when the processed original images include original images with long exposure, normal exposure and short exposure, the dynamic range and image details of the image can be adjusted so that the content presented by the obtained captured image is more realistic and the image quality is relatively better.

[0194] It should be noted that the above sequence of S141 to S146 is only one example. When performing environmental detection, you can also first perform illumination detection on the scene to be shot to determine whether the detected illumination is less than the illumination threshold, and then perform dynamic range detection on the scene to be shot to determine whether the detected dynamic range value is less than the dynamic range threshold, or the order of the two can be swapped. The specific execution process can be adjusted as needed. In addition, other steps can be added or some steps can be deleted on this basis, and the embodiment of the present application does not impose any restrictions on this.

[0195] S15, such as Figure 3As shown, the level 1 processing frame number is processed using the level 1 processing algorithm to obtain the corresponding captured image.

[0196] In combination with the example in S14 above, when the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, the determined A1 algorithm is used to process the a1 frame in the original image collected corresponding to the first shooting time to obtain the corresponding captured image.

[0197] When the illumination is less than the illumination threshold and the dynamic range is less than the dynamic range threshold, the determined B1 algorithm is used to process the b1 frame in the original image collected corresponding to the first shooting time to obtain the corresponding captured image.

[0198] When the illumination is greater than or equal to the illumination threshold and the dynamic range value is also greater than or equal to the dynamic range threshold, the determined C1 algorithm is used to process the c1 frame in the original image collected corresponding to the first shooting time to obtain the corresponding captured image.

[0199] When the illumination is greater than or equal to the illumination threshold and the dynamic range is less than the dynamic range threshold, the determined D1 algorithm is used to process the d1 frame in the original image collected corresponding to the first shooting time to obtain the corresponding captured image.

[0200] For example, when the first operation is the first photo-taking time, an original image is collected correspondingly, and a total of 9 frames of original images are collected. When the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, the determined A1 algorithm can be used to process 6 of the 9 frames (at this time a1=6) to obtain the corresponding captured images.

[0201] When the illumination is less than the illumination threshold and the dynamic range is less than the dynamic range threshold, the determined B1 algorithm can be used to process 5 frames (in this case b1=5) out of the 9 frames to obtain corresponding captured images.

[0202] When the illumination is greater than or equal to the illumination threshold and the dynamic range value is also greater than or equal to the dynamic range threshold, the determined C1 algorithm can be used to process 4 frames (c1=4 at this time) of the 9 frames to obtain corresponding captured images.

[0203] When the illumination is greater than or equal to the illumination threshold and the dynamic range is less than the dynamic range threshold, the determined D1 algorithm can be used to process 3 frames (at this time d1=3) out of the 9 frames to obtain corresponding captured images.

[0204] It should be understood that the present application can determine different scenes by performing environmental detection on the scene to be shot, and then, based on the different subdivided scenes, select different algorithms, and select different numbers of frames from the corresponding collected original images for processing, so as to adaptively improve the quality and effect of the captured images corresponding to the first number of photos in each scene.

[0205] S16. When the first operation corresponds to the xth photo-taking number, determine the memory usage, where x is an integer greater than 1.

[0206] It should be understood that the memory usage refers to the amount of data stored in the memory. Since the camera will increase the memory usage each time it collects an original image or obtains a frame of captured image, that is, the memory usage will increase relatively.

[0207] Then, as the number of photos taken increases, the memory usage will become larger and larger. In this way, when the memory usage reaches a certain level, the relevant technology may fail to respond to the photo command. In order to avoid this situation, this application will monitor the memory usage in real time. Different processing algorithms and / or different processing frames can be set for different memory usage to reduce the pressure of memory increment. Among them, the memory usage can be obtained by obtaining data inside the electronic device.

[0208] S17. Determine the x-level processing frame number and the x-level processing algorithm corresponding to the number of photos taken according to the memory usage.

[0209] Among them, the x-level processing algorithm is used to refer to the algorithm used in the processing corresponding to the x-th number of photos, and the x-level processing frame number is used to refer to the number of frames of the original image used in the processing corresponding to the x-th number of photos. For example, when x=2, the second number of photos corresponds to the 2nd level of processing frame number and the 2nd level of processing algorithm; when x=3, the third number of photos corresponds to the 3rd level of processing frame number and the 3rd level of processing algorithm; the names of subsequent processing frame numbers and processing algorithms are deduced accordingly and will not be repeated.

[0210] For example, Fig. 9 A schematic diagram of a process for determining the corresponding number of processing frames and a processing algorithm according to memory usage is shown.

[0211] like Fig. 9 As shown, the above S17 may include the following S171 to S175.

[0212] S171. For the x-th photo-taking number, determine whether the memory usage is less than a first memory threshold.

[0213] S172: When the memory usage is less than the first memory threshold, determine the x-level processing algorithm as the first photographing algorithm, and determine the x-level processing frame number as the first frame number.

[0214] The first frame number refers to all or part of the frames in the original image collected corresponding to the xth shooting number.

[0215] S173: When the memory usage is greater than or equal to the first memory threshold, continue to determine whether the memory usage is less than the second memory threshold.

[0216] S174: When the memory usage is less than the second memory threshold, determine the x-level processing algorithm as the second photographing algorithm, and determine the x-level processing frame number as the second frame number.

[0217] The second frame number refers to all or part of the frames in the original image collected corresponding to the xth shooting number.

[0218] S175. When the memory usage is greater than or equal to the second memory threshold, determine that the x-level processing algorithm is a third photographing algorithm, and determine that the x-level processing frame number is a third frame number.

[0219] The third frame number refers to all or part of the frames in the original image collected corresponding to the xth shooting number.

[0220] It should be understood that the first memory threshold is less than the second memory threshold, and the second memory threshold is less than the total memory. Here, the size of the first memory threshold and the second memory threshold can be set and modified as needed, and the embodiment of the present application does not impose any limitation on this.

[0221] It should be understood that the first photographing algorithm, the second photographing algorithm and the third photographing algorithm may be the same or different.

[0222] As an example, when the first photographing algorithm, the second photographing algorithm and the third photographing algorithm are the same, the first frame number, the second frame number and the third frame number gradually decrease.

[0223] As another example, when the first photographing algorithm, the second photographing algorithm and the third photographing algorithm are different, the processing time of the first photographing algorithm, the second photographing algorithm and the third photographing algorithm gradually decreases. At this time, the sizes of the first frame number, the second frame number and the third frame number can be the same, or gradually decrease.

[0224] Among them, in the above two cases, the reduction amount of the second frame number relative to the first frame number and the reduction amount of the third frame number relative to the second frame number can be the same or different, and can be set and modified as needed. The embodiment of the present application does not impose any restrictions on this.

[0225] In the second case above, the processing steps included in the first photographing algorithm, the second photographing algorithm, and the third photographing algorithm are gradually reduced, or the complexity of the processing steps included is gradually reduced, so that the processing time is gradually reduced. The processing time reduced by the second photographing algorithm relative to the first photographing algorithm and the processing time reduced by the third photographing algorithm relative to the second photographing algorithm can be the same or different, and can be specifically set and modified as needed, and the embodiment of the present application does not impose any restrictions on this.

[0226] In the present application, the first frame number, the second frame number, or the third frame number can be selected from all frames of the original image collected by the corresponding number of shots in the acquisition order for processing, or the first frame number, the second frame number, or the third frame number can be arbitrarily extracted from all frames of the original image for processing, or the images collected by one or more cameras can be selected as the first frame number, the second frame number, or the third frame number images for processing in a specified camera manner. Of course, the method of selecting the first frame number, the second frame number, or the third frame number can be the same or different, and can be specifically set and modified as needed, and the embodiment of the present application does not impose any restrictions on this.

[0227] In addition, the original image of the first frame number selected may include at least one of a long-exposure original image, a normal-exposure original image, and a short-exposure original image, and the original image of the second frame number selected may include at least one of a long-exposure original image, a normal-exposure original image, and a short-exposure original image. Similarly, the original image of the third frame number selected may also include a long-exposure original image, a normal-exposure original image, and a short-exposure original image. Wherein, regardless of whether the first frame number, the second frame number, and the third frame number are the same, the number of long-exposure original images and short-exposure original images included in the first frame number, the second frame number, and the third frame number may be gradually reduced. In this way, when the memory usage increases, the dynamic range will be abandoned to ensure that the user's need for fast photography is met as much as possible.

[0228] It should be understood that in the above S171 to S175, only two memory thresholds are set, namely the first memory threshold and the second memory threshold. Through the two memory thresholds, the memory size can be divided into three intervals. When the memory occupancy satisfies one of the intervals, the algorithm and frame number corresponding to the interval are used for processing to obtain the corresponding captured image. In addition, a memory threshold can also be set to divide the memory size into two intervals; or, three, four or more memory thresholds can be set to subdivide the memory size into multiple intervals. The specific settings and modifications can be made as needed, and the embodiments of the present application do not impose any restrictions.

[0229] Here, the number of x-level processing frames corresponding to the x-th number of photographs can be the same as the number of processing frames corresponding to the 1st number of photographs, or can be less than the number of processing frames corresponding to the 1st number of photographs; the x-level processing algorithm corresponding to the x-th number of photographs can be the same as the 1-level processing algorithm corresponding to the 1st number of photographs, or can have a shorter processing time than the 1-level processing algorithm corresponding to the 1st number of photographs. The specific settings and modifications can be made as needed, and the embodiments of the present application do not impose any restrictions on this.

[0230] When the first photographing algorithm, the second photographing algorithm and the third photographing algorithm are the same, the first photographing algorithm, the second photographing algorithm and the third photographing algorithm may be the same as the level 1 processing algorithm or have a shorter processing time than the level 1 processing algorithm. In this case, the first frame number, the second frame number and the third frame number gradually decrease, and the first frame number may be the same as the level 1 processing frame number or have a shorter processing time than the level 1 processing frame number, and then the second frame number and the third frame number are both shorter than the level 1 processing frame number.

[0231] When the processing time of the first photographing algorithm, the second photographing algorithm, and the third photographing algorithm gradually decreases, the first photographing algorithm may be the same as the level 1 processing algorithm corresponding to the first photographing number, or the processing time of the first photographing algorithm is less than the level 1 processing algorithm corresponding to the first photographing number, then the processing time of the second photographing algorithm and the third photographing algorithm is less than the level 1 processing algorithm. In this case, when the first frame number, the second frame number, and the third frame number are the same, the first frame number, the second frame number, and the third frame number are the same as or less than the level 1 processing frame number. When the first frame number, the second frame number, and the third frame number gradually decrease, the first frame number is the same as or less than the level 1 processing frame number, then the second frame number and the third frame number are less than the level 1 processing frame number.

[0232] S18, using an x-level processing algorithm to process the x-level processing frames to obtain a corresponding captured image.

[0233] Fig.10 A schematic diagram showing the growth of memory usage is shown.

[0234] like Fig.10 As shown, in combination with the example in S17 above, when x=2, the memory usage corresponding to the second number of photos is determined. If the memory usage corresponding to the second number of photos is less than the first memory threshold, the level 2 processing algorithm is determined to be the first photo algorithm, and the level 2 processing frame number is determined to be the first frame number.

[0235] When x=3, the memory usage corresponding to the third number of photos is determined. If the memory usage corresponding to the third number of photos is greater than the first memory threshold but less than the second memory threshold, the level 3 processing algorithm is determined to be the second photo algorithm, and the level 3 processing frame number is determined to be the second frame number.

[0236] When x=4, the memory usage corresponding to the fourth number of photos is determined. If the memory usage corresponding to the fourth number of photos is greater than the first memory threshold but less than the second memory threshold, the level 4 processing algorithm is determined to be the second photo algorithm, and the level 4 processing frame number is determined to be the second frame number. In this way, the processing of the fourth number of photos is equivalent to the processing of the third number of photos.

[0237] When x=5, the memory usage corresponding to the fifth photo taking number is determined. If the memory usage corresponding to the fifth photo taking number is greater than the second memory threshold, the level 5 processing algorithm is determined to be the third photo taking algorithm, and the level 5 processing frame number is determined to be the third frame number. The same is true for the subsequent steps, which will not be described in detail here.

[0238] It should be understood that based on dividing the memory size into multiple interval ranges, when the memory usage is larger and the higher interval range is satisfied, the processing algorithm is the same, then the number of processed frames is gradually reduced, or, if the number of processed frames is the same, the processing time of the processing algorithm is reduced; or, both the processing time and the number of processed frames of the processing algorithm are reduced. At this time, the growth rate of the memory usage will slow down rapidly, reducing the pressure of memory increment in electronic devices.

[0239] In this way, compared with the prior art method in which each photo is processed with a single processing algorithm and processing frame number, or the method in which the memory usage increases at a fixed rate, the present application can timely adjust the processing algorithm and processing frame number according to the memory usage. When the memory is larger, the growth rate will decrease and tend to be flat, so that the electronic device can achieve more photos under the condition of the same memory size.

[0240] It should also be understood that Fig.10 The amount of memory corresponding to the first number of photographs is not shown in the figure. At this time, the original image collected by the first number of photographs is processed according to the above-mentioned environmental detection results. The method of processing according to the result of memory usage is applicable to the second and subsequent number of photographs. Of course, the same method as the second and subsequent number of photographs can also be selected for processing the first number of photographs, and the embodiment of the present application does not limit this.

[0241] For example, x is 1, and the memory occupancy corresponding to the first photo-taking number is determined. When the memory occupancy is less than the first memory threshold, the level 1 processing algorithm is determined to be the first photo-taking algorithm, and the level 1 processing frame number is determined to be the first frame number; when the memory occupancy is greater than or equal to the first memory threshold but less than the second memory threshold, the level 1 processing algorithm is determined to be the second photo-taking algorithm, and the level 1 processing frame number is determined to be the second frame number; when the memory occupancy is greater than the second memory threshold, the level 1 processing algorithm is determined to be the third photo-taking algorithm, and the level 1 processing frame number is determined to be the third frame number; then, the level 1 processing frame number is processed using the determined level 1 processing algorithm to obtain the corresponding captured image.

[0242] S19: Save the captured image.

[0243] The captured images can be stored in the gallery or displayed.

[0244] In combination with the above process, in response to one click operation, one frame of captured image can be obtained, and in response to multiple click operations, multiple frames of captured images can be obtained; the specific number of captured images is related to the number of user operations, and the embodiment of the present application does not impose any restrictions on this.

[0245] The embodiment of the present application provides a photographing method, in response to a first operation of a user on a first control, an electronic device uses a camera to collect an original image. When the first operation corresponds to the first photographing number, an environmental detection is performed for the scene to be photographed, and then, the first-level processing frame number and the first-level processing algorithm corresponding to the first photographing number are determined according to the environmental detection result. When the first operation corresponds to the second photographing number and subsequent photographing numbers, the memory usage is first detected, the determined memory usage size is compared with a preset memory threshold, and then, according to different memory usage sizes, different processing frames and processing algorithms are selected.

[0246] When processing the first number of photos, since the scene is subdivided according to the environmental detection results, different algorithms and frame rates are selected according to different scenes, so that the quality and effect of the captured images corresponding to the first number of photos in each scene can be adaptively improved.

[0247] When processing the second photo and subsequent photos, since the memory usage is subdivided, different algorithms and frame rates are selected according to different memory usage. When the memory usage is larger, the algorithm with shorter processing time and / or the number of processing frames is reduced, thereby reducing the pressure of memory increment, improving processing efficiency, and meeting the needs of fast photo taking.

[0248] When taking photos or other operations, electronic devices will gradually generate high heat, which will cause the device temperature of the electronic device to gradually rise. If you take photos quickly and continuously, the device temperature may rise exponentially. In this way, in the case of overheating, the electronic device will not be able to operate normally and will not respond to the photo command.

[0249] In this regard, the embodiments of the present application may also detect the device temperature of the electronic device before processing the collected data, and switch the processing algorithm and the number of processing frames in case of overheating to reduce the workload and reduce the incremental heat pressure.

[0250] Fig.11 FIG. 2 shows a flow chart of another method for taking photos. Fig.11 As shown, this method Figure 3 On the basis of, the following S20 and S21 may also be included.

[0251] The contents of S11 to S19 can refer to the above Figure 3 The description will not be repeated here.

[0252] S20: After collecting an original image once in response to a first operation, when the first operation corresponds to a second photo-taking number or other subsequent photo-taking numbers, determining whether the device temperature is less than a temperature threshold.

[0253] The device temperature refers to the temperature inside the electronic device. Usually, as the electronic device works, it will generate a certain amount of heat, which will cause the device temperature to increase. The device temperature can be collected by a temperature sensor inside the electronic device.

[0254] S21, when the device temperature is greater than or equal to the temperature threshold, one frame of the original image collected this time is processed using a single-frame algorithm to obtain a corresponding captured image, and then the obtained captured image is saved.

[0255] The temperature threshold can be set and modified as needed, and the present application embodiment does not impose any restrictions on this. The device temperature can be obtained by obtaining data inside the electronic device.

[0256] It should be understood that the single-frame algorithm refers to an algorithm that processes one frame of original image. The single-frame algorithm is different from the level 1 processing algorithm. The processing time of the single-frame algorithm is shorter than that of the level 1 processing algorithm (A1 algorithm, B1 algorithm, C1 algorithm or D1 algorithm).

[0257] Here, the single-frame algorithm may include processing steps such as default pixel correction (DPC), RAW domain noise reduction (RAWNF), black level correction (BLC), lens shading correction (LSC), automatic white balance, demosaicing, color correction matrix (CCM), YUV domain noise reduction, tone mapping, gamma correction, color space conversion, color enhancement (CE), etc., with fast processing speed and short time consumption.

[0258] Bad pixel correction, bad pixels are white dots in the output image under a completely dark environment, and black dots in the output image under a bright environment. Generally, the three primary color channel signals should have a linear response relationship with the ambient brightness, but due to the poor signal output by the image sensor, white dots or black dots may appear. For this, bad pixels can be automatically detected and repaired, or a bad pixel link list can be established to repair bad pixels at fixed positions. Among them, a point refers to a pixel.

[0259] Noise reduction refers to the process of reducing the noise in an image. Common methods include mean filtering, Gaussian filtering, bilateral filtering, etc. RAW domain noise reduction refers to the process of reducing the noise in RAW domain images. YUV domain noise reduction refers to the process of reducing the noise in YUV domain images.

[0260] Black level correction is due to the existence of dark current in the image sensor, which causes the pixel to have a certain output voltage when there is no light, and pixels at different positions may correspond to different output voltages. Therefore, it is necessary to correct the output voltage corresponding to the pixel when there is no light (i.e., black).

[0261] Lens shading correction can solve the problem of shadows appearing around the lens due to uneven refraction of light by the lens.

[0262] Automatic white balance is designed to eliminate the impact of light sources on image sensor imaging, simulate the color constancy of human vision, and ensure that the white seen in any scene is truly white. Therefore, it is necessary to correct the color temperature and automatically adjust the white balance to the appropriate position.

[0263] De-mosaicing, since each pixel in the RAW domain image only corresponds to the color information of one channel, the surrounding pixel information can be used to estimate other colors. For example, through linear interpolation, the color information of the other two channels missing for each pixel can be determined, thereby restoring all channel information of all pixels in the image. The process of de-mosaicing is equivalent to converting the image from the RAW domain to the RGB domain. Here, de-mosaicing can also be called color interpolation.

[0264] Color correction: Since the image captured by the camera is somewhat different from the color people expect, color correction is needed. Since the automatic white balance has already calibrated the white color, other colors besides white can be calibrated through color correction.

[0265] Tone mapping refers to mapping all pixels in an image with a spatially invariant mapping function. That is, when tone mapping transforms the dynamic range of an image, each pixel in the image uses the same transformation function, which is a one-to-one mapping relationship. Here, tone mapping can also be called dynamic range compression (DRC).

[0266] Gamma correction refers to editing the gamma curve of an image in a non-linear way to edit the image's tones, detect the dark and light parts of the image, and increase them proportionally, thereby improving the image contrast effect.

[0267] Color space conversion refers to converting an image from the RGB domain to the YUV domain. Thus, the image obtained after the previous step can be converted from the RGB domain to the YUV domain to reduce the amount of data subsequently stored and transmitted, saving bandwidth.

[0268] Color enhancement is to make the original unsaturated color information saturated and rich. Here, color enhancement can also be called color processing.

[0269] The above is only an example. The single-frame algorithm may also delete or add other steps, and the level 1 processing algorithm (A1 algorithm, B1 algorithm, C1 algorithm or D1 algorithm) may include other steps on the basis of the single-frame algorithm. Of course, the level 1 processing algorithm (A1 algorithm, B1 algorithm, C1 algorithm or D1 algorithm) may also be different from the steps included in the single-frame algorithm, and the embodiments of the present application do not impose any restrictions on this.

[0270] It should be understood that since the single-frame algorithm can be processed only in the image signal processor, the processing speed of the single-frame algorithm is relatively fast and the processing time is relatively short.

[0271] When the first photographing algorithm, the second photographing algorithm and the third photographing algorithm are the same, the processing time of the single-frame algorithm is shorter than that of the third photographing algorithm. Thus, the processing time of the second photographing algorithm is also shorter than that of the first photographing algorithm.

[0272] When the processing times of the first photographing algorithm, the second photographing algorithm and the third photographing algorithm gradually decrease, the single-frame algorithm may be the same as the third photographing algorithm, or the processing time of the single-frame algorithm may be relatively shorter than that of the third photographing algorithm.

[0273] In combination with the above, when the device temperature is higher than the temperature threshold, the single-frame algorithm can be switched to process 1 frame of the original image according to the above steps, so as to minimize the workload of the electronic device, slow down the growth of heat, and reduce the incremental pressure of heat, so that the electronic device can achieve more photos before it fails to work, meeting the user's photography needs. When the device temperature is lower than the temperature threshold, the memory usage of the electronic device can be judged according to the process described in S16 to S19 above, and different algorithms and different frame numbers can be switched according to different memory usages to reduce the pressure of memory increment, so that more photos can be taken before the memory is full to meet the user's photography needs.

[0274] It should be noted that Fig.11 The execution order shown is only an example. After capturing an original image, when determining that the first operation is the first photo shooting, the device temperature of the electronic device may be detected. Then, when the device temperature of the electronic device is greater than or equal to the temperature threshold, a single-frame algorithm is used to process one frame in the original image. When the device temperature of the electronic device is less than the temperature threshold, an environmental detection is performed on the scene to be shot. Alternatively, the device temperature detection and the environmental detection of the scene to be shot may be performed simultaneously, and the number of level 1 processing frames and the level 1 processing algorithm corresponding to the first photo shooting may be determined based on the device temperature detection results and the environmental detection results. Alternatively, the temperature detection may be performed for each photo shooting to determine whether it is less than the temperature threshold, and then the subsequent related steps may be performed.

[0275] It should be understood that the above method only sets one temperature threshold, and the present application can also set multiple or three or more temperature thresholds, which can be set specifically as needed. Exemplarily, two temperature thresholds can be set, namely a first temperature threshold and a second temperature threshold, and the first temperature threshold is less than the second temperature threshold. Through the two temperature thresholds, the temperature range can be divided into three intervals. When the device temperature meets one of the intervals, the algorithm and frame number corresponding to the interval are used for processing to obtain the corresponding captured image. On this basis, different memory thresholds can also be set for different temperature intervals to execute different judgment processes.

[0276] Fig.12 FIG. 2 shows a flow chart of another photographing method. When two temperature thresholds are set, namely, a first temperature threshold and a second temperature threshold, the method Fig.11 Based on the above, a new process including S22 is provided, which is introduced below. Fig.12 As shown, S22 may also include the following S221 to S231.

[0277] S221. After capturing an original image once in response to a first operation, when the first operation corresponds to an xth number of photographs (the second number of photographs or other subsequent numbers of photographs), determine whether the device temperature is less than a first temperature threshold.

[0278] S222: When the device temperature is lower than the first temperature threshold, continue to determine whether the memory usage is lower than the first memory threshold.

[0279] S223. According to S222, when the device temperature is less than the first temperature threshold and the memory usage is also less than the first memory threshold, determine that the x-level processing algorithm is the first photographing algorithm, and determine that the x-level processing frame number is the first frame number.

[0280] S224: Using a first photographing algorithm, process the first number of frames in the original image collected this time to obtain a corresponding photographed image.

[0281] S225. According to S222, when the device temperature is less than the first temperature threshold, but the memory usage is greater than or equal to the first memory threshold, the second photographing algorithm may be determined to be an x-level processing algorithm, and the x-level processing frame number may be determined to be a second frame number.

[0282] S226: Use a second photographing algorithm to process a second number of frames in the original image collected this time to obtain a corresponding photographed image.

[0283] S227. In the above S221, when the device temperature is greater than or equal to the first temperature threshold, continue to determine whether the device temperature is less than the second temperature threshold.

[0284] S228: When the device temperature is greater than or equal to the first temperature threshold but less than the second temperature threshold, it may be further determined whether the memory usage is less than the second memory threshold.

[0285] At this time, when the device temperature is less than the second temperature threshold and the memory usage is less than the second memory threshold, it means that the device temperature is not particularly high and the memory is not full. It is possible to continue processing with an algorithm with a longer processing time or with original images with a larger number of frames, such as executing S225 and S226 above.

[0286] S229. According to S228, when the device temperature is greater than or equal to the first temperature threshold and less than the second temperature threshold, but the memory usage is greater than or equal to the second memory threshold, it can be determined that the third photo-taking algorithm is an x-level processing algorithm, and the x-level processing frame number is determined to be the third frame number.

[0287] S230: Using a third photographing algorithm, process a third number of frames in the original image collected this time to obtain a corresponding photographed image.

[0288] S231. According to S227, when it is determined that the device temperature is greater than or equal to the first temperature threshold and also greater than the second temperature threshold, it indicates that the device temperature is very high. Therefore, a single-frame algorithm can be used to process one frame of the original image collected this time to obtain a corresponding captured image.

[0289] It should be understood that when the temperature range is divided into multiple intervals, and the memory size is divided into multiple intervals, the temperature and memory conditions can be combined. In this way, when the temperature is low and the memory usage is small, the algorithm with a long processing time is used to process multiple frames in the original image to ensure good image quality and effect; when the temperature is low and the memory usage is large or the temperature is high and the memory usage is low, adaptive adjustments are made to use an algorithm with a relatively short processing time to process more frames in the original image to balance the relationship between image quality and temperature and memory usage; when the temperature is high and the memory usage is high, the algorithm with a shorter processing time is switched to process fewer frames in the original image to reduce the pressure of temperature increment and memory increment, and ensure normal photography with a slight reduction in image quality; in addition, when the temperature is particularly high, regardless of the memory usage, the simplest algorithm is directly switched to process the least frames in the original image. Although the image quality is reduced at this time, the temperature increment is reduced to the greatest extent to ensure response to the user's photography command and meet the user's photography needs.

[0290] It should be understood that the above is only an example, and multiple temperature thresholds and memory thresholds can be set to distinguish different situations. They can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0291] The above method provided by the present application can make a judgment based on the device temperature and memory usage of the electronic device at the time of the current trigger when the user triggers the photo command, and then associate the determined processing algorithm and the number of processing frames with the number of photos. When the user triggers the command at a low frequency and the background processing of the electronic device is relatively timely, the number of processing frames can be processed according to the determined processing algorithm to obtain the corresponding captured image. When the background of the electronic device cannot process in time, the determined processing algorithm and the number of processing frames are associated with the number of photos, and then stored together with the collected original image and other data, and then wait for the electronic device to call and process it later.

[0292] However, when the user suddenly increases the frequency of triggering the photo command, the data obtained by the electronic device surges, causing serious data congestion. In this case, when the data corresponding to a certain number of photos is subsequently called and processed, since some time has passed between the processing time and the response time, the memory usage of the electronic device has changed greatly during this period of time. At this time, the originally determined processing algorithm will no longer be applicable and needs to be adaptively adjusted.

[0293] Alternatively, when the data corresponding to a certain number of photos is called for processing, the memory usage is close to the memory limit. If the processing is still performed according to the previously determined processing algorithm, it will inevitably cause an unresponsive situation. At this time, it is also necessary to adaptively adjust the processing algorithm in time to reduce the memory increment again.

[0294] In response to the above two situations, the embodiment of the present application provides a photo-taking method to switch the processing algorithm to adapt to the memory situation at the actual processing moment.

[0295] Fig.13 Another photographing process provided by the embodiment of the present application. Fig.13 As shown, the method 30 may include the following 31 to S39.

[0296] S31. The electronic device displays a first interface, where the first interface includes a first control.

[0297] S32: When a first operation of the user on the first control is detected, in response to the first operation, the electronic device uses a camera to collect an original image.

[0298] S33: When the first operation corresponds to the xth photo-taking time, determine the memory usage.

[0299] S34. Determine the x-level processing frame number and the x-level processing algorithm corresponding to the number of photos taken, with x being an integer greater than 1, based on the memory usage.

[0300] Among them, the process from S31 to S34 is the same as the above Figure 3The processes of S11, S12, S16 and S17 are the same, and you can refer to the above description, which will not be repeated here.

[0301] S35: Determine whether the frequency at which the user performs the first operation on the first control meets a preset frequency condition.

[0302] The frequency may indicate the speed at which the user performs the first operation, such as the speed at which the user clicks the capture button. For example, the interval between two consecutive capture commands issued by the user is detected. If the interval is less than the second preset interval for 10 or 20 consecutive times, it can be said that the frequency meets the preset frequency condition. The second preset interval can be set as needed, for example, 100m or 200ms. The second preset interval is less than the first preset interval.

[0303] It should be understood that when the frequency does not meet the preset frequency condition, it means that the number of times the user triggers the photo command is relatively mild, and the user takes multiple normal photos; when the frequency does not meet the preset frequency condition, it means that the number of times the user triggers the photo command is relatively large, and the user expects to take multiple photos quickly. The preset frequency condition, that is, the number of consecutive detections of the interval time length of adjacent photo times, and the second preset interval time length can be set and modified as needed, and the embodiment of the present application does not impose any restrictions on this.

[0304] Optionally, in the embodiment of the present application, it is also possible to first determine whether the frequency of the user performing the first operation on the first control meets the preset frequency condition, and then, when the preset frequency condition is met, determine the memory usage, according to Figure 3 The method shown determines the x-level processing frame number and the x-level processing algorithm corresponding to the number of photographs.

[0305] As an example, when the frequency meets the preset frequency condition, the memory usage is determined according to Figure 3 When the method shown determines the x-level processing frame number and the x-level processing algorithm corresponding to the number of photographs, the x-level processing frame number may not include the long-exposure original image and the short-exposure original image, and the x-level processing algorithm does not include the high dynamic range (HDR) algorithm.

[0306] It should be understood that when the frequency meets the preset frequency condition, it means that the user suddenly and quickly issued intensive photo-taking commands. At this time, the long-exposure original images and short-exposure original images included in the x-level processing frames can be removed, and the x-level processing algorithm can be switched to a non-HDR algorithm. In this way, by giving up the protection of the dynamic range and switching to a non-HDR algorithm that only processes the normal exposure original images, the subsequent processing efficiency can be improved to meet the user's fast photo-taking needs.

[0307] S36: Determine whether the difference between the memory waterline and the memory usage is less than a preset difference.

[0308] Among them, the memory waterline refers to the maximum usage of memory. The size of the memory waterline can be obtained by obtaining parameters provided by the manufacturer, or it may need to be set and modified. The embodiment of the present application does not impose any restrictions on this.

[0309] It should be understood that when the difference between the memory waterline and the memory usage is greater than the preset difference, it means that the memory usage is still far from reaching the level of the memory waterline, and there is still a lot of available storage space; when the difference between the memory waterline and the memory usage is less than the preset difference, it means that the memory usage is very close to reaching the level of the memory waterline, and there is little available storage space left. The size of the preset difference can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this.

[0310] S37. When the frequency meets the preset frequency condition or the difference between the memory waterline and the memory usage is less than the preset difference, determine the number of photos waiting in the background, and use the x-level processing algorithm corresponding to each waiting number of photos as the initial processing algorithm.

[0311] Determining the number of photos waiting in the background refers to determining which photo times are waiting in the background. The number of photos waiting in the background includes 1, two, or more. For example, it includes 2, namely the 5th level photo number and the 6th level photo number. Then, the 5th level processing algorithm corresponding to the 5th photo number is used as the initial processing algorithm corresponding to itself, and the 6th level processing algorithm corresponding to the 6th photo number is used as the initial processing algorithm corresponding to itself. Of course, there may be no waiting photo times in the background. When there are no waiting photo times, there is no need to perform subsequent steps.

[0312] It should be understood that when the frequency meets the preset frequency condition, it means that the user has increased the frequency of triggering the photo command; when the difference between the memory waterline and the memory usage is less than the preset difference, it means that the memory is approaching the upper limit. In these two cases, it is necessary to switch the processing algorithm, and when switching, it is necessary to first clarify the processing algorithm that needs to be replaced.

[0313] The above are only two example conditions, and other conditions can also be set to start the replacement process, or the above conditions can be cancelled, and the secondary processing algorithm is replaced when processing the number of photos waiting in the background. The specific settings and adjustments can be made as needed, and this application does not impose any restrictions on this.

[0314] S38, taking the x-level processing algorithm corresponding to the most recently determined number of photographing times as the target processing algorithm, and using the target processing algorithm to replace the initial processing algorithm corresponding to the waiting number of photographing times.

[0315] It should be understood that the most recent number of photographs refers to the number of photographs currently recorded in response to the user's first operation. The process of determining the processing algorithm and processing frame number corresponding to the most recent number of photographs can be determined using the method described in S17 above, which will not be repeated here.

[0316] Since the processing algorithm and number of processing frames corresponding to the number of photos waiting in the background are determined based on the previous memory usage, the previous memory usage is different from the current memory usage. Therefore, the currently determined memory usage can be used as a consideration to replace all the processing algorithms corresponding to the number of photos waiting in the background with the currently determined processing algorithms to adapt to the current memory situation.

[0317] Optionally, before replacement, the processing algorithm corresponding to the most recently determined number of photographs can be compared with the initial processing algorithm corresponding to the number of photographs to be waited for. If they are the same, no replacement is required; if they are different, replacement is performed. Thus, by comparing the processing algorithms, replacement is performed only for cases where the processing algorithms are different, which can avoid invalid replacement and reduce workload.

[0318] Optionally, the number of processing frames corresponding to the number of photos to be waited for can be used as the initial number of processing frames, and the number of processing frames corresponding to the most recently determined number of photos to be taken can be used as the target number of processing frames, and then the initial number of processing frames corresponding to the number of photos to be waited for can be replaced by the target number of processing frames.

[0319] It should be understood that when the currently determined memory usage is taken into consideration, the number of processed frames corresponding to the number of photos waiting in the background can be completely replaced by the target number of processed frames.

[0320] Before replacement, the number of processed frames corresponding to the most recently determined number of photographs can be compared with the number of processed frames corresponding to the waiting number of photographs. If they are the same, no replacement is required; if they are different, replacement is performed. Thus, by comparing the number of processed frames, replacement is performed only when the number of frames is different, thus avoiding invalid replacement and reducing workload.

[0321] Among them, according to Figure 3From the above content, it can be seen that when the number of processing frames corresponding to the number of photos is determined for the first time, the number of processing frames is the same, or, as the memory usage increases, the number of processing frames gradually decreases. It can be seen that when the number of processing frames corresponding to the number of photos is determined for the second time, if replacement is required, it must be excluded that the case where the number of processing frames is the same, and the previous larger number of processing frames is replaced with the most recent smaller number of processing frames, which is equivalent to reducing the number of processing frames corresponding to the number of photos to be waited for each time. The smaller the number of processing frames after replacement, the higher the work efficiency when determining the captured images later, and the less the memory usage increases.

[0322] It should be noted that when the target processing frame number is used to replace the initial processing frame number, only the value of the processing frame number is changed, and the corresponding original image is not replaced. For example, the number of photos to be waited for is the 10th photo number. When responding to the first operation, 9 frames of original images are collected. When determining the number of processing frames, 5 frames of original images are selected as the processing frames, and "5 frames" is the initial processing frame number. If it is subsequently determined that the target processing frame number is 1 frame, the initial processing frame number "5 frames" can be replaced with "1 frame", and the original image corresponding to the 1 frame is the 1 frame selected from the original 5 frames of original images.

[0323] It should also be understood that after the above replacement method is started, it can be executed repeatedly. In other words, the processing algorithm and / or processing frame number corresponding to the waiting number of photos can be replaced once each time a new photo command is responded to. In other words, for the processing algorithm and / or processing frame number corresponding to a certain waiting number of photos, the processing algorithm and / or processing frame number can be replaced multiple times before being used for processing to obtain the corresponding captured image.

[0324] Therefore, the processing algorithm corresponding to the number of photos waiting in the background is the processing algorithm determined according to the memory usage in response to the user's first operation on the first control, or is the target processing algorithm after the last replacement during the waiting process.

[0325] For example, when the most recently determined number of photographs is the 20th number of photographs, the processing algorithm corresponding to the 20th number of photographs is used as the target processing algorithm, and the corresponding number of processing frames is used as the target number of frames; then, the processing algorithms corresponding to the multiple waiting number of photographs can be used as the initial processing algorithm, and the corresponding number of processing frames can be used as the initial processing frame number, and then the initial processing algorithm is replaced with the target processing algorithm, and the initial processing frame number is replaced with the target processing frame number.

[0326] In response to the user's first operation again, the number of photos corresponding to the most recent one is updated to the 21st photo number, and the processing algorithm corresponding to the 21st photo number is used as the target processing algorithm, and the corresponding number of processing frames is used as the target number of processing frames. Then, the processing algorithms corresponding to the waiting multiple photo numbers (the target processing algorithm corresponding to the 20th photo number) can be used as the initial processing algorithm, and the corresponding number of processing frames (the target number of processing frames corresponding to the 20th photo number) can be used as the initial processing frame number, and then the initial processing algorithm is replaced with the target processing algorithm for the 21st photo number, and the initial processing frame number is replaced with the target number of processing frames for the 21st photo number. If the user continues to perform the first operation, the subsequent replacement process is analogous, and will not be repeated here.

[0327] S39, when any one of the photographing times waiting in the background is called for processing, the x-level processing frames in the original image collected by this photographing time are processed by using the corresponding target processing algorithm to obtain the corresponding photographed image.

[0328] In addition, when the frequency does not meet the preset frequency condition, or the difference is greater than or equal to the preset difference, the above steps do not need to be performed, and the processing continues according to the originally determined processing algorithm. For example, as shown in S18, the x-level processing frame number is processed using the originally determined x-level processing algorithm to obtain the corresponding captured image. Alternatively, although the above-mentioned frequency or difference conditions are met, when there is no waiting number of photos in the background, the above steps do not need to be performed.

[0329] S40: Save the captured image.

[0330] The captured images can be stored in the gallery or displayed.

[0331] The above-mentioned photographing method provided by the embodiment of the present application, in response to the user's first operation on the first control, the electronic device uses the camera to collect an original image once, and when the frequency used to perform the first operation on the first control meets the preset frequency condition or the difference between the memory waterline and the memory occupancy is less than the preset difference, the number of photos waiting in the background is determined and the processing algorithm corresponding to each number of photos waiting is used as the initial processing algorithm corresponding to each. Then, the processing algorithm corresponding to the number of photos determined most recently is used as the target processing algorithm, and the initial processing algorithm corresponding to the number of photos waiting is replaced by the target processing algorithm. In this way, when the number of photos waiting is subsequently called for processing, the replaced target processing algorithm can be used to process the initial processing frame number in the original image collected by the number of photos, so as to obtain the corresponding captured image. Since the processing algorithm corresponding to the number of photos determined most recently is determined according to the current memory situation, the processing algorithm of the number of photos waiting is adjusted accordingly, which can control the degree of memory accumulation, reduce the memory occupancy, and realize continuous and fast photography.

[0332] In addition, the number of processing frames can also be replaced. When the number of processing frames decreases with the increase in the number of photos, the number of processing frames corresponding to the most recently determined number of photos is used as the target number of processing frames, and the initial number of processing frames corresponding to the number of photos to be waited is replaced by the target number of processing frames. After the replacement, the number of processing frames corresponding to the number of photos to be waited is reduced, so that when the number of photos to be waited is called for processing, the reduced target number of processing frames can be used to determine the captured image. Since the number of processing frames corresponding to the most recently determined number of photos is determined based on the current memory situation, the number of processing frames for the number of photos to be waited is adjusted accordingly, which can also control the degree of memory accumulation and reduce memory usage.

[0333] On this basis, if the number of photos waiting in the background takes a long time to be processed, the corresponding processing algorithm and / or processing frame number can be replaced multiple times as the number of photos determined most recently changes, and the latest target processing algorithm can be used to replace the last target processing algorithm, and / or the latest target processing frame number can be used to replace the last target processing frame number, so that when the number of photos waiting in the background is actually processed, the latest target processing algorithm can be used for processing. In this way, after multiple replacements, the latest data can be used as a benchmark to reduce the memory usage as much as possible, and more continuous and fast photos can be achieved.

[0334] Compared with only determining the processing algorithm and processing frame number once before, this method performs secondary replacement, or even multiple replacements, and can update and optimize the processing algorithm and / or processing frame number in real time, more flexibly adapt to the current memory situation, and ensure that photo taking can proceed normally.

[0335] The above generally introduces the process of the photographing method provided by the present application, which will be further illustrated by using an embodiment below.

[0336] Example 1

[0337] Fig.14 A table provided in an embodiment of the present application is used to illustrate the processing algorithm and processing frame number corresponding to each number of photo taking times.

[0338] Combination Figure 4 As shown, when the user opens the camera application, the mobile phone displays a shooting interface, which includes a shooting button. When the user takes a photo, he or she clicks the shooting button multiple times with his or her finger.

[0339] When the mobile phone detects the user's first click operation, in response to the first click operation, the mobile phone uses the camera to capture an original image, for example, 9 frames of original images in the RAW domain are captured, and the number of photos taken this time can be recorded as the first number of photos taken.

[0340] Since this click operation corresponds to the first photo shooting, the mobile phone can perform environmental detection for the scene to be photographed, such as illumination detection and dynamic range detection. At this time, it is also determined whether the detected illumination is less than the illumination threshold and whether the detected dynamic range value is less than the dynamic range threshold based on the environmental detection result.

[0341] When the illumination is less than the illumination threshold, the dynamic range value is less than the dynamic range threshold, or when the scene to be captured is a low-brightness, low-dynamic-range scene, such as Fig.14 As shown, it can be determined that the first-level processing algorithm is the A1 algorithm, and the A1 algorithm is, for example, the first night scene algorithm. In addition, it can be determined that the number of first-level processing frames corresponding to the first shooting time is, for example, 6 frames, and the 6 frames refer to 6 frames of original images selected from the 9 frames of original images collected corresponding to the first shooting time.

[0342] When the mobile phone detects the user's second click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 9 frames of original images, and the number of photos taken this time can be recorded as the second number of photos taken.

[0343] Since this photo number corresponds to the second photo number, x is greater than 1, so the mobile phone continues to determine the memory usage. Since the memory usage at this time is less than the first memory threshold, it can be determined that the second-level processing algorithm corresponding to the second photo number is the first photo algorithm, and the first photo algorithm is also, for example, the first night scene algorithm. At the same time, it can be determined that the second-level processing frame number corresponding to the second photo number is 6 frames, and 6 frames is the first frame number. The 6 frames refer to the 6 frames of original images selected from the 9 frames of original images collected corresponding to the second photo number.

[0344] When the mobile phone detects the user's third click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 9 frames of original images, and the number of photos taken this time can be recorded as the third number of photos taken.

[0345] Since this photo number corresponds to the third photo number, x is greater than 1, so the mobile phone determines the memory usage at this time, and the memory usage at this time is greater than the first memory threshold but less than the second memory threshold. Therefore, it can be determined that the 3rd level processing algorithm corresponding to the third photo number is the second photo algorithm, and the preset second photo algorithm is the same as the first photo algorithm, that is, the second photo algorithm is the first night scene algorithm. At the same time, it can be determined that the 3rd level processing frame number corresponding to the third photo number is 5 frames, 5 frames are the second frame number, and the second frame number is less than the first frame number. The 5 frames refer to the 5 frames of original images selected from the 9 frames of original images collected corresponding to the third photo number.

[0346] When the mobile phone detects the user's fourth click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 9 frames of original images, and the number of photos taken this time can be recorded as the fourth number of photos taken.

[0347] Since this photo number corresponds to the 4th photo number, x is greater than 1, so the mobile phone determines the memory usage at this time, and the memory usage at this time is greater than the second memory threshold. Therefore, it can be determined that the 4th level processing algorithm corresponding to the 4th photo number is the third photo algorithm, and the preset third photo algorithm is the same as the second photo algorithm and the first photo algorithm, that is, the third photo algorithm is the first night scene algorithm. At the same time, it can be determined that the 4th level processing frame number corresponding to the 4th photo algorithm is 4 frames, 4 frames is the third frame number, and the third frame number is less than the second frame number. The 4 frames refer to the 4 frames of original images selected from the 9 frames of original images collected corresponding to the 4th photo number.

[0348] The processing of subsequent photo taking times is similar and will not be described in detail here.

[0349] For example, the first night scene algorithm may include all the steps included in the single-frame algorithm, and a night scene algorithm module generated based on the Unet network model, and the night scene algorithm module may fuse multiple frames of original images in the RAW domain into one frame of image in the RAW domain. Of course, the night scene algorithm module may also be generated based on other models, and the first night scene algorithm may also include other modules or steps, and the embodiments of the present application have no limitation on this.

[0350] In addition, the reduction amount of the number of processing frames corresponding to each number of photo taking can be the same or different, and can be set and modified according to needs. The embodiments of the present application do not impose any restrictions on this.

[0351] In the above embodiment, when the scene to be photographed is a low-brightness, low-dynamic-range scene, it means that the ambient light in the scene to be photographed is relatively dark. If a general photography algorithm is used for processing, the obtained photographed image will be unclear, which will cause the user to not be able to see the details in the photographed image. In this regard, in the embodiment of the present application, the first night scene algorithm can be selected for processing to increase the brightness of the image during the processing, so that the subsequently obtained photographed image can present more details.

[0352] On this basis, based on dividing the memory size into multiple interval ranges, during multiple photo taking processes, as the memory usage increases, the number of processing frames determined for the number of photos corresponding to different memory ranges gradually decreases. At this time, although the selected processing algorithm is the same, during subsequent processing, due to the decrease in the number of processing frames, the processing time of the processing algorithm will be correspondingly shortened, and the growth rate of the memory usage will tend to be flat. Therefore, in the embodiment of the present application, under multiple photo taking times, as the number of photos increases, based on the same algorithm, the number of processing frames corresponding to each photo taking time is reduced, thereby reducing the processing time of the processing algorithm, improving processing efficiency, and at the same time slowing down the growth rate of memory usage and reducing the pressure of memory increment.

[0353] Example 2

[0354] Fig.15 Another table provided for an embodiment of the present application is used to illustrate the processing algorithm and processing frame number corresponding to each number of photographs.

[0355] Combination Figure 4 As shown, when the user opens the camera application, the mobile phone displays a shooting interface, which includes a shooting button. When the user takes a photo, he or she clicks the shooting button multiple times with his or her finger.

[0356] When the mobile phone detects the user's first click operation, in response to the click operation, the mobile phone uses the camera to capture an original image, for example, 6 frames of original images in the RAW domain are captured. The number of photos taken this time can be recorded as the first number of photos taken.

[0357] Since this click operation corresponds to the first photo shooting, the mobile phone can perform environmental detection for the scene to be photographed, for example, illumination detection and dynamic range detection. At this time, it is also determined whether the detected illumination is less than the illumination threshold and whether the detected dynamic range value is less than the dynamic range threshold based on the environmental detection result.

[0358] When the illumination is greater than the illumination threshold and the dynamic range value is less than the dynamic range threshold, or when the scene to be captured is a high-brightness, low-dynamic range scene, such as Fig.15As shown, it can be determined that the first-level processing algorithm is the D1 algorithm, and the D1 algorithm is the first multi-frame noise reduction (MFNR) algorithm. In addition, it can be determined that the number of first-level processing frames corresponding to the first shooting number is 6 frames, and the 6 frames refer to all 6 frames of original images collected corresponding to the first shooting number.

[0359] When the mobile phone detects the user's second click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 6 frames of original images, and the number of photos taken this time can be recorded as the second number of photos taken.

[0360] Since this photo number corresponds to the second photo number, x is greater than 1, so the mobile phone can then detect the device temperature to determine whether the device temperature is less than the temperature threshold. If it is less than the temperature threshold, it continues to determine the memory usage. Since the memory usage at this time is less than the first memory threshold, it can be determined that the second-level processing algorithm corresponding to the second photo number is the first photo algorithm, and the first photo algorithm is, for example, the second MFNR algorithm. At the same time, it can be determined that the second-level processing frame number corresponding to the second photo number is 6 frames, and 6 frames is the first frame number. The 6 frames refer to all 6 frames of original images collected corresponding to the second photo number.

[0361] When the mobile phone detects the user's third click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 6 frames of original images, and the number of photos taken this time can be recorded as the third number of photos taken.

[0362] Since this photo shooting number corresponds to the third photo shooting number, x is greater than 1, so the mobile phone can then detect the device temperature to determine whether the device temperature is less than the temperature threshold. If it is less than the temperature threshold, the memory usage at this time is determined. The memory usage at this time is greater than the first memory threshold but less than the second memory threshold. Therefore, it can be determined that the 3rd-level processing algorithm corresponding to the third photo shooting number is the second photo shooting algorithm. The preset second photo shooting algorithm is different from the first photo shooting algorithm, and the processing time of the second photo shooting algorithm is shorter than the processing time of the first photo shooting algorithm. The second photo shooting algorithm is, for example, the third MFNR algorithm. At the same time, it can be determined that the 3rd-level processing frame number corresponding to the third photo shooting number is 6 frames, 6 frames is the second frame number, and the second frame number is the same as the first frame number. The 6 frames refer to all 6 frames of original images collected corresponding to the third photo shooting number.

[0363] When the mobile phone detects the user's fourth click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 6 frames of original images, and the number of photos taken this time can be recorded as the fourth number of photos taken.

[0364] Since this photo number corresponds to the 4th photo number, and x is greater than 1, the device temperature can be detected to determine whether the device temperature is less than the temperature threshold. If it is less than the temperature threshold, the mobile phone continues to determine the memory usage at this time. The memory usage at this time is greater than the second memory threshold. Therefore, it can be determined that the 4th level processing algorithm corresponding to the 4th photo number is the third photo algorithm. The preset third photo algorithm is different from the second photo algorithm and the first photo algorithm, and the processing time of the third photo algorithm is shorter than the processing time of the second photo algorithm. The third photo algorithm is, for example, the fourth MFNR algorithm. At the same time, it can be determined that the 4th level processing frame number corresponding to the 4th photo number is 6 frames, and 6 frames is the third frame number, which is the same as the second frame number and the first frame number. The 6 frames refer to all 6 frames of original images collected corresponding to the 4th photo number.

[0365] When the mobile phone detects the user's fifth click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, or 6 frames of original images, and the number of photos taken can be recorded as the fifth number of photos taken.

[0366] Since this photo shooting number corresponds to the fifth photo shooting number, x is greater than 1, so the device temperature can be detected to determine whether the device temperature is less than the temperature threshold. If it is greater than the temperature threshold, there is no need to judge the memory usage, and the processing algorithm at this time is directly determined to be a single-frame algorithm, and the number of processing frames is 1 frame. This 1 frame refers to 1 frame of the original image among the 6 frames of original images collected corresponding to the fifth photo shooting number.

[0367] The processing of subsequent photo taking times is similar and will not be described in detail here.

[0368] Among them, for example, the first MFNR algorithm can be a process of first aligning multiple frames in the original image, and then processing the aligned original image using a wavelet fusion algorithm; the second MFNR algorithm can be an optical flow algorithm, and the third MFNR algorithm and the fourth MFNR algorithm are other algorithms. The embodiments of the present application have no restrictions on this.

[0369] In the above embodiment, when the scene to be photographed is a scene with high brightness and low dynamic range, it means that the ambient light in the scene to be photographed is strong, but the dynamic range is small. At this time, if a general photography algorithm is used for processing, the obtained captured image has large noise and small dynamic range. In this regard, in the embodiment of the present application, the MFNR algorithm can be selected for processing to reduce the noise of the captured image and improve the dynamic range of the captured image.

[0370] On this basis, based on dividing the memory size into multiple interval ranges, during multiple photo taking processes, as the memory usage increases, the processing time of the processing algorithm determined by the number of photos taken in different memory ranges gradually decreases. At this time, although the selected number of processing frames is the same, during subsequent processing, the processing time of the processing algorithm gradually becomes shorter, making the growth rate of the memory usage tend to be flat. Therefore, in the embodiment of the present application, under multiple photo taking times, as the number of photos increases, based on the same number of processing frames, the processing time or performance of the processing algorithm corresponding to each photo taking time can be reduced, thereby improving processing efficiency, while slowing down the growth rate of memory usage and reducing the pressure of memory increment.

[0371] In addition, when the device temperature is higher than the temperature threshold, it can directly switch to a single-frame algorithm to process one frame of the original image to reduce the workload, slow down the increase in heat, and reduce the incremental heat pressure, so that the mobile phone can take more photos and meet the user's photo needs.

[0372] Example 3

[0373] Fig.16 Another table provided in an embodiment of the present application is used to illustrate the processing algorithm and processing frame number corresponding to each number of photo taking times. Fig.17 for Fig.16 Schematic diagram of the corresponding processing process.

[0374] Combination Figure 5 As shown, when a user opens a camera application to record a video, the mobile phone displays a video recording interface, which includes a snapshot control. The user uses a finger to click the snapshot control multiple times while recording the video.

[0375] When the mobile phone detects the user's first click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, for example, 9 frames of original images are collected, and the 9 frames of original images include 6 frames of normal exposure original images, 1 frame of long exposure original image and 2 frames of short exposure original images. The number of photos taken this time can be recorded as the first number of photos taken.

[0376] Since this click operation corresponds to the first photo shooting, the mobile phone performs environmental detection for the scene to be photographed, for example, performs illumination detection and dynamic range detection. At this time, it is also determined whether the detected illumination is less than the illumination threshold value and whether the detected dynamic range value is less than the dynamic range threshold value based on the detection results.

[0377] When the illumination is greater than the illumination threshold, the dynamic range value is greater than the dynamic range threshold, or when the scene to be captured is a high-brightness, high-dynamic range scene, such as Fig.16As shown in (a), it can be determined that the level 1 processing algorithm is the C1 algorithm, and the C1 algorithm is the first HDR algorithm. In addition, it can be determined that the level 1 processing frame number corresponding to the first shooting time is 9 frames, and the 9 frames refer to all 9 frames of original images collected corresponding to the first shooting time.

[0378] The first HDR algorithm may include an HDR algorithm module, and the electronic device may implement the processing process of the first HDR algorithm through the HDR algorithm module.

[0379] For example, the first HDR algorithm module is generated based on multi-exposure fusion processing and tone mapping model, and can fuse multiple frames of original images with different exposure values ​​or different exposure levels into one frame of image in the RAW domain and perform color mapping transformation processing on it. Among them, the tone mapping model can be any one of the Unet network model, the Resnet network model and the Hdrnet network model. Of course, the tone mapping model can also be other models, and the embodiments of the present application do not impose any restrictions on this.

[0380] At this time, combined Fig.17 As shown, the background starts to process the first level processing frames using the first HDR algorithm determined for the first number of photos, that is, all 9 frames of original images collected corresponding to the first number of photos are processed. This process can be called the first photo processing.

[0381] When the mobile phone detects the user's second click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, which is still 9 frames of original images, including 7 frames of normal exposure original images, 1 frame of long exposure original image and 1 frame of short exposure original image. The number of photos taken this time can be recorded as the second number of photos taken.

[0382] Since the number of photos taken this time corresponds to the second number of photos taken, and x is greater than 1, the mobile phone can then determine whether the frequency of the user's click operation on the snapshot control meets the preset frequency condition. If the frequency does not meet the preset frequency condition, the memory usage is continued to be determined. Since the memory usage at this time is less than the first memory threshold, it can be determined that the second-level processing algorithm corresponding to the second number of photos is the first photo algorithm. The first photo algorithm is, for example, the second HDR algorithm, and the second HDR algorithm is different from the first HDR algorithm. At the same time, it can be determined that the second-level processing frame number corresponding to the second number of photos is 8 frames, and 8 frames is the first number of frames. The 8 frames refer to 5 frames of normally exposed original images, 1 frame of long-exposed original images, and 2 frames of short-exposed original images among the 9 frames of original images collected corresponding to the second number of photos. Alternatively, the 8 frames refer to 5 frames of normally exposed original images among the 9 frames of original images collected corresponding to the second number of photos, and 1 frame of long-exposed original images and 2 frames of short-exposed original images collected by multiplexing the first number of photos. Alternatively, the 8 frames refer to 1 frame of long-exposure original image and 2 frames of short-exposure original images captured by the first photo-taking time, and 5 frames of normal-exposure original images whose acquisition time is close to the long-exposure original image or the short-exposure original image.

[0383] It should be noted that when the user clicks to take photos very quickly, that is, when the frequency of triggering the photo command is relatively high, the content difference between the images is relatively small. Therefore, at this time, it is possible to choose to partially reuse or fully reuse the original image collected by the first photo shooting. In this way, the electronic device does not need to re-frame, which reduces the workload and memory usage.

[0384] The subsequent process of determining the number of photos taken from the third to the fifth time is similar to the process of determining the number of photos taken from the second time, and will not be repeated here.

[0385] When the mobile phone detects the user's sixth click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, which is still 9 frames of original images, including 6 frames of normal exposure original images, 1 frame of long exposure original image and 2 frames of short exposure original images. This number of photos can be recorded as the sixth number of photos.

[0386] Since this photo shooting number corresponds to the sixth photo shooting number, x is greater than 1, so the mobile phone can then determine whether the frequency of the user's click operation on the snapshot control meets the preset frequency condition. If the frequency does not meet the preset frequency condition, the memory usage at this time is continued to be determined. The memory usage at this time is greater than the first memory threshold, but less than the second memory threshold. Therefore, it can be determined that the 6-level processing algorithm corresponding to the sixth photo shooting number is the second photo shooting algorithm, the preset second photo shooting algorithm is different from the first photo shooting algorithm, and the processing time of the second photo shooting algorithm is shorter than the processing time of the first photo shooting algorithm. The second photo shooting algorithm is, for example, the third HDR algorithm, which is different from the first HDR algorithm and the second HDR algorithm. At the same time, it can be determined that the 6-level processing frame number corresponding to the sixth photo shooting number is 7 frames, 7 frames is the second frame number, and the second frame number is smaller than the first frame number. The 7 frames refer to 4 frames of normally exposed original images, 1 frame of long-exposed original images, and 2 frames of short-exposed original images among the 9 frames of original images collected corresponding to the sixth photo shooting number. Alternatively, the 7 frames refer to 4 frames of normally exposed original images among the 9 frames of original images collected corresponding to the 6th shooting number, and 1 frame of long-exposure original image and 2 frames of short-exposure original images collected by the 1st shooting number. Alternatively, the 7 frames refer to 1 frame of long-exposure original image and 2 frames of short-exposure original images collected by the 1st shooting number, and 5 frames of normally exposed original images collected at a closer time to the long-exposure original image or the short-exposure original image.

[0387] Combination Fig.17 As shown, when the user performs the second to sixth click operations, since the first photo processing is still in progress, only all the original images corresponding to the second to sixth photos, as well as the determined processing algorithms at each level, the number of processing frames at each level and other related data can be stored, and the background waiting state is entered.

[0388] Then, when the first photo processing is completed, the background waits for the second to sixth photos according to the principle of time from near to far, and first processes the data related to the sixth photo. For the sixth photo, the background starts to process the 7 frames of original images using the determined third HDR algorithm, that is, the 4 frames of normally exposed original images, 1 frame of long-exposed original image, and 2 frames of short-exposed original images collected corresponding to the sixth photo are processed. This process can be called the second photo processing.

[0389] Next, while processing the data related to the sixth photo-taking number, when the mobile phone detects the seventh click operation of the user, in response to the click operation, the mobile phone uses the camera to collect an original image, which is still 9 frames of original images, and the 9 frames of original images include 6 frames of normal exposure original images, 1 frame of long exposure original image and 2 frames of short exposure original images. This photo-taking number can be recorded as the seventh photo-taking number.

[0390] Since this photo number corresponds to the 7th photo number, x is greater than 1, so the mobile phone can then determine whether the frequency of the user's click operation on the snapshot control meets the preset frequency condition. If the frequency meets the preset frequency condition, it is necessary to abandon the protection of the dynamic range and switch to a non-HDR algorithm that only processes the normally exposed original image. For example, the memory usage at this time can be determined. If the memory usage at this time is greater than the first memory threshold but less than the second memory threshold, the 7-level processing algorithm corresponding to the 7th photo number can be switched from the third HDR algorithm to the third MFNR algorithm. At the same time, it can be determined that the 7-level processing frame number corresponding to the 7th photo number is 4 frames of normally exposed original images, and other long-exposure original images and short-exposure original images are removed.

[0391] In addition, since the frequency meets the preset frequency condition, it can also be determined that the number of photos waiting in the background at this time is the second to fifth photo taking times. Based on this, the processing algorithms corresponding to the second to fifth photo taking times stored before can be used as the initial processing algorithm, and the 7-level processing algorithm determined by the seventh photo taking time can be used as the target processing algorithm, and then the initial processing algorithm corresponding to the waiting number of photos can be replaced by the target processing algorithm.

[0392] like Fig.16 As shown in (b), the 7-level processing algorithm determined by the 7th shooting number is the third MFNR algorithm. Then, the third MFNR algorithm is the target processing algorithm. Fig.16 As shown in (a), the second HDR algorithm originally corresponding to the second shooting number is the initial processing algorithm. After the initial processing algorithm is replaced by the target processing algorithm, the second-level processing algorithm corresponding to the second shooting number will be changed to the third MFNR algorithm. The replacement process from the third to the fifth shooting number is similar and will not be repeated here.

[0393] Combination Fig.17 As shown, when the user performs the 7th click operation, since the 2nd photo processing is still in progress, only all the original images corresponding to the 7th photo number, as well as the determined 7-level processing algorithm, 7-level processing frame number and other related data can be stored, and the system enters the background waiting state.

[0394] When the mobile phone detects the user's 8th click operation, in response to the click operation, the mobile phone uses the camera to collect an original image, which is still 9 frames of original images. The 9 frames of original images include 6 frames of normal exposure original images, 1 frame of long exposure original image and 2 frames of short exposure original images. The number of photos taken this time can be recorded as the 8th number of photos taken.

[0395] Since this photo number corresponds to the 8th photo number, x is greater than 1, so the mobile phone can then determine whether the frequency of the user's click operation on the snapshot control meets the preset frequency condition. If the frequency meets the preset frequency condition, it is necessary to abandon the protection of the dynamic range and switch to a non-HDR algorithm that only processes the normally exposed original image. For example, the memory usage at this time can be determined. If the memory usage at this time is greater than the second memory threshold, the 8-level processing algorithm corresponding to the 8th photo number can be switched from the third HDR algorithm to the fourth MFNR algorithm, and the fourth MFNR algorithm is different from the third MFNR algorithm. At the same time, it can be determined that the 8-level processing frame number corresponding to the 8th photo number is 3 frames of normally exposed original images, and other long-exposed original images and short-exposed original images are removed.

[0396] In addition, since the frequency meets the preset frequency condition, it can also be determined that the number of photos waiting in the background at this time is the 2nd to 5th photo number, and the 7th photo number. Based on this, the processing algorithm after the last replacement of the 2nd to 5th photo number can be used as the initial processing algorithm, and the 7-level processing algorithm determined by the 7th photo number can be used as the initial processing algorithm, and the 8-level processing algorithm determined by the 8th photo number can be used as the target processing algorithm, and then the target processing algorithm can be used to replace the initial processing algorithm corresponding to the waiting number of photos.

[0397] Combination Fig.16 As shown in (c), the 8th level processing algorithm determined by the 8th shooting number is the fourth MFNR algorithm, so the fourth MFNR algorithm is the target processing algorithm. The third MFNR algorithm originally corresponding to the second shooting number is the initial processing algorithm. After the initial processing algorithm is replaced by the target processing algorithm, the 2nd level processing algorithm corresponding to the second shooting number is changed to the fourth MFNR algorithm. The replacement process of the 3rd to 5th and 7th shooting times is similar and will not be repeated here.

[0398] Combination Fig.17 As shown, when the user performs the 8th click operation, since the 2nd photo processing is still in progress, only all the original images corresponding to the 8th photo number, as well as the determined 8-level processing algorithm, 8-level processing frame number and other related data can be stored, and the system enters the background waiting state.

[0399] Then, when the user performs the 9th to 20th click operations, the processing algorithm corresponding to the number of photos waiting in the background can continue to be replaced according to the x-level processing algorithm corresponding to the most recently determined number of photos as the target processing algorithm. When processing the number of photos waiting in the background, it can be processed according to the processing algorithm after the last replacement.

[0400] In the above embodiments, the present application changes the processing algorithm and the number of frames for taking photos in different situations by combining multiple factors such as the environmental detection results in the scene to be photographed, the frequency of user operations, the memory usage, etc.

[0401] On this basis, since the processing cannot be performed in time when the number of photos is large, the present application will also promptly replace the processing algorithm of the waiting number of photos with the processing algorithm determined most recently. In this way, the processing can be based on the latest data, adapting to the current memory situation as much as possible, reducing memory increments, and achieving more continuous and fast photos.

[0402] The above is a detailed introduction to the photographing method provided in the embodiment of the present application. The following is an introduction on how a user enables the photographing method provided in the embodiment of the present application in conjunction with the display interface of the electronic device.

[0403] Fig.18 A schematic diagram of a display interface of an electronic device provided in an embodiment of the present application.

[0404] Exemplarily, in response to a user's click operation, when the electronic device 100 runs a camera application, the electronic device 100 displays the following Fig.18 The shooting interface shown in (a) in FIG. A navigation bar is displayed on the top of the interface, wherein an icon indicating "AI photography" is displayed in the middle.

[0405] In response to the user's click operation on "AI Photography", the electronic device 100 displays the following Fig.18 The prompt box shown in (b) in the figure displays a prompt message that "AI photography" has been turned on. At this time, the electronic device 100 can enable the program related to the photographing method provided in the embodiment of the present application when shooting.

[0406] It should be understood that the above is only an example of a user enabling the photographing method provided in the embodiment of the present application from the display interface of an electronic device. Of course, the photographing method provided in the embodiment of the present application can also be enabled in other ways, or the photographing method provided in the embodiment of the present application can be directly used by default during the shooting process. The embodiment of the present application does not impose any restrictions on this.

[0407] Fig.19A schematic diagram of a display interface of another electronic device provided in an embodiment of the present application.

[0408] Combination Fig.18 and Fig.19 As shown, Fig.19 As shown in (a) in the figure, when "AI Photography" is turned on, if the user opens the camera application and quickly and continuously clicks the shooting button in the shooting interface multiple times, during the clicking process, when the background uses the shooting method provided by the embodiment of the present application to process, when the memory usage increases, a processing algorithm with shorter processing time and / or fewer processing frames will be used for processing. Therefore, the quality of the captured image processed subsequently will be reduced to a certain extent. In this regard, Fig.19 As shown in (b) in the figure, in order to prompt the user that the image quality has changed, a prompt message "The current image quality may be reduced" can be displayed above the preview image in the shooting interface.

[0409] Of course, the above is only an example, and other prompts may be displayed to remind the user, or no prompts may be displayed, or the user may be reminded in other ways, and the embodiments of the present application do not impose any restrictions on this.

[0410] Combination of the above Figures 1 to 19 The photographing method and the related display interface provided by the embodiment of the present application are described in detail; Figure 20 to Figure 23 The electronic devices, devices and chips provided by the embodiments of the present application are described in detail. Fig. 20 A hardware system of an electronic device suitable for the present application is shown.

[0411] The electronic device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device 100.

[0412] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0413] It should be noted that Fig. 20 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include Fig. 20 More or fewer components than those shown, or the electronic device 100 may include Fig. 20 Combinations of some of the components shown, or the electronic device 100 may include Fig. 20 Subassemblies of some of the components shown. Fig. 20 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0414] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated devices. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0415] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0416] Exemplarily, the processor 110 can be used to execute the photographing method of the embodiment of the present application; display a first interface, which includes a first control; when a first operation on the first control is detected, in response to the first operation, collect an original image; for the x-th number of photographs, determine the memory usage; when the memory usage is less than the first memory threshold, determine that the x-level processing algorithm is the first photographing algorithm, and determine the x-level processing frame number to be the first frame number, where x is an integer greater than 1; when the memory usage is greater than or equal to the first memory threshold but less than the second memory threshold, determine that the x-level processing algorithm is the second photographing algorithm, and determine the x-level processing frame number to be the second frame number; when the memory usage is greater than the second memory threshold, determine that the x-level processing algorithm is the third photographing algorithm, and determine the x-level processing frame number to be the third frame number, where the first memory threshold is less than the second memory threshold; use the x-level processing algorithm to process the x-level processing frame number in the original image to obtain a corresponding captured image; and save the captured image.

[0417] Fig. 20 The connection relationship between the modules shown is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of multiple connection modes in the above embodiments.

[0418] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0419] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 is also used to detect parameters such as battery capacity, battery cycle number, and battery health status (leakage, impedance).

[0420] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0421] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0422] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0423] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0424] The camera 193 is used to capture static images or videos. It can be triggered to start by application instructions to realize the photo function, such as taking pictures of any scene. The camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the camera angle (also known as the scene to be photographed, the target scene, or the scene image that the user expects to shoot) to form an image; the filter is mainly used to filter out the excess light waves in the light (for example, light waves other than visible light, such as infrared); the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, etc. format. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

[0425] Among them, the camera 193 can be located on the front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to needs, and this application does not impose any restrictions.

[0426] Exemplarily, the electronic device 100 includes a front camera and a rear camera. For example, the front camera or the rear camera may include one or more cameras. Take the electronic device 100 having four rear cameras as an example. In this way, when the electronic device 100 starts the four rear cameras for shooting, the photographing method provided in the embodiment of the present application can be used. Alternatively, the camera is arranged on an external accessory of the electronic device 100, and the external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is any angle between 0-360 degrees. For example, when the electronic device 100 takes a selfie, the external accessory drives the camera to rotate to a position facing the user. Of course, when the mobile phone has multiple cameras, only some of the cameras may be arranged on the external accessories, and the remaining cameras may be arranged on the electronic device 100 body. The embodiment of the present application does not impose any restrictions on this.

[0427] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0428] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0429] The internal memory 121 may also store software codes of the photographing method provided in the embodiment of the present application. When the processor 110 runs the software codes, the process steps of the photographing method are executed to achieve fast and continuous photographing.

[0430] The internal memory 121 may also store captured images.

[0431] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing files such as music in the external memory card.

[0432] Of course, the software code of the photographing method provided in the embodiment of the present application can also be stored in an external memory, and the processor 110 can run the software code through the external memory interface 120 to execute the process steps of the photographing method to obtain multiple frames of photographed images. The photographed images obtained by the electronic device 100 can also be stored in an external memory.

[0433] It should be understood that the user can specify whether to store the image in the internal memory 121 or the external memory. For example, when the electronic device 100 is connected to the external memory, if the electronic device 100 captures a frame of image, a prompt message may pop up to prompt the user to store the image in the external memory or the internal memory; of course, there may be other specifying methods, and the embodiment of the present application does not impose any restrictions on this; or, when the electronic device 100 detects that the memory capacity of the internal memory 121 is less than the preset amount, the image may be automatically stored in the external memory.

[0434] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0435] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0436] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting.

[0437] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0438] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0439] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0440] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0441] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0442] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0443] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0444] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0445] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0446] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0447] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0448] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects.

[0449] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0450] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting the SIM card interface 195 or removing the SIM card from the SIM card interface 195 .

[0451] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0452] The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is introduced below.

[0453] Fig.21 It is a schematic diagram of a software system of an electronic device provided in an embodiment of the present application.

[0454] like Fig.21As shown, the system architecture may include an application layer (APP) 210 , an application framework layer 220 , a hardware abstract layer (HAL) 230 , a driver layer 240 and a hardware layer 250 .

[0455] The application layer 210 may include a camera application or other applications, and other applications include but are not limited to: camera, gallery and other applications.

[0456] The application layer 210 is at the top of the entire framework and is responsible for interacting directly with the user. Once it receives a direct or indirect request from the user, such as taking a photo, it sends the request to the application framework layer 220 through the interface and waits for the application framework layer 220 to return the processing result, which includes image data and camera parameters, etc. The application layer 210 then feeds back the result to the user.

[0457] The application framework layer 220 may provide an application programming interface (API) and a programming framework to the application programs of the application layer; the application framework layer may include some predefined functions.

[0458] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera equipment; wherein the camera management may be used to provide an access interface for managing the camera; and the camera equipment may be used to provide an interface for accessing the camera.

[0459] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may include camera device 1, camera device 2, etc.; the camera hardware abstraction layer may be connected to a camera algorithm library, and the camera hardware abstraction layer may call algorithms in the camera algorithm library.

[0460] In the present application, a perception engine for performing various detections may be provided in a hardware abstraction layer.

[0461] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver, a digital signal processor driver, and a graphics processor driver.

[0462] The hardware layer 250 may include a plurality of image sensors, a plurality of image signal processors, a digital signal processor, a graphics processor, and other hardware devices.

[0463] For example, the hardware layer 250 includes sensors and image signal processors; the sensors may include sensor 1, sensor 2, depth sensor (time of flight, TOF), multi-spectral sensor, etc. The image signal processor may include image signal processor 1, image signal processor 2, etc.

[0464] In the present application, by calling the hardware abstraction layer interface in the hardware abstraction layer 230, the connection between the application layer 210 and the application framework layer 220 above the hardware abstraction layer 230 and the driver layer 240 and the hardware layer 250 below can be realized to realize camera data transmission and function control.

[0465] In the camera hardware interface layer in the hardware abstraction layer 230, manufacturers can customize functions according to their needs. Compared with the hardware abstraction layer interface, the camera hardware interface layer is more efficient, flexible, and has low latency, and can also call ISP and GPU more richly to achieve image processing. The image input into the hardware abstraction layer 230 can come from an image sensor or a stored picture.

[0466] The scheduling layer in the hardware abstraction layer 230 includes a general functional interface for implementing management and control.

[0467] The camera service layer in the hardware abstraction layer 230 is used to access the interface of the ISP and other hardware.

[0468] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0469] The camera application in the application layer can be displayed as an icon on the screen of the electronic device 100. When the icon of the camera application is clicked by the user to be triggered, the electronic device 100 starts to run the camera application. When the camera application runs on the electronic device 100, the camera application calls the interface corresponding to the camera application in the application framework layer 210, and then starts the camera driver by calling the hardware abstraction layer 230, turns on the camera 193 on the electronic device 100, and at the same time, the camera algorithm library starts to load the photo-taking method used in the embodiment of the present application.

[0470] Then, after one or more frames of raw images are captured by the image sensor, the captured raw images can be processed by the image signal processor and returned to the hardware abstraction layer, and processed using a processing algorithm called from the camera algorithm library to generate captured images, which are then saved and / or transmitted to the display screen for display.

[0471] The following introduces an image processing device 300 provided in an embodiment of the present application for implementing the above-mentioned photographing method. Fig. 22is a schematic diagram of an image processing device 300 provided in an embodiment of the present application.

[0472] like Fig. 22 As shown, the image processing device 300 includes a display unit 310 , an acquisition unit 320 and a processing unit 330 .

[0473] The display unit 310 is used to display a first interface, and the first interface includes a first control.

[0474] The acquisition unit 320 is used to detect a first operation of a user on a first control.

[0475] The processing unit 330 is configured to collect an original image once in response to the first operation.

[0476] The processing unit 330 is also used to determine the memory occupancy for the x-th photo-taking number; when the memory occupancy is less than the first memory threshold, determine that the x-level processing algorithm is the first photo-taking algorithm, and determine the x-level processing frame number as the first frame number, where x is an integer greater than 1; when the memory occupancy is greater than or equal to the first memory threshold but less than the second memory threshold, determine that the x-level processing algorithm is the second photo-taking algorithm, and determine the x-level processing frame number as the second frame number; when the memory occupancy is greater than the second memory threshold, determine that the x-level processing algorithm is the third photo-taking algorithm, and determine the x-level processing frame number as the third frame number, and the first memory threshold is less than the second memory threshold; use the x-level processing algorithm to process the x-level processing frame number in the original image to obtain the corresponding captured image; and save the captured image.

[0477] It should be noted that the above-mentioned image processing device 300 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0478] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit, and / or other suitable components that support the described functions.

[0479] Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0480] The embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium runs on the image processing device 300, the image processing device 300 executes the photographing method shown above.

[0481] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0482] The embodiment of the present application also provides a computer program product including computer instructions, which, when executed on the image processing device 300, enables the image processing device 300 to execute the photographing method shown above.

[0483] Fig.23 A schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.23 The chip shown can be a general-purpose processor or a dedicated processor. The chip includes a processor 401. The processor 401 is used to support the image processing device 300 to execute the technical solution shown above.

[0484] Optionally, the chip further includes a transceiver 402, and the transceiver 402 is used to accept the control of the processor 401, and to support the image processing device 300 to execute the technical solution shown above.

[0485] Optional, Fig.23 The chip shown may further include: a storage medium 403 .

[0486] It should be noted that Fig.23The chip shown can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits that can perform the various functions described throughout this application.

[0487] The electronic device, image processing device 300, computer storage medium, computer program product, and chip provided in the above-mentioned embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding beneficial effects of the methods provided above, and will not be repeated here.

[0488] It should be understood that the above is only to help those skilled in the art to better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application. According to the above examples given, those skilled in the art can obviously make various equivalent modifications or changes. For example, some steps in each embodiment of the above detection method may be unnecessary, or some steps may be newly added. Or a combination of any two or any multiple embodiments of the above. Such modifications, changes or combined schemes also fall within the scope of the embodiments of the present application.

[0489] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0490] It should also be understood that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0491] It should also be understood that in the embodiments of the present application, "pre-setting" and "pre-definition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including an electronic device), and the present application does not limit its specific implementation method.

[0492] It should also be understood that the division of the methods, situations, categories and embodiments in the embodiments of the present application is only for the convenience of description and should not constitute a special limitation. The features of various methods, categories, situations and embodiments can be combined without contradiction.

[0493] It should also be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0494] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Display a first interface, wherein the first interface includes a first control; In response to a first operation on the first control, capturing an original image; In response to the first operation, determining a memory usage; When the memory usage is less than a first memory threshold, a first frame number of images in the original image is processed using a first photographing algorithm to obtain a corresponding photographed image; When the memory usage is greater than or equal to the first memory threshold and less than a second memory threshold, a second frame number image in the original image is processed by a second photographing algorithm to obtain a corresponding photographed image; When the memory usage is greater than the second memory threshold, using a third photographing algorithm to process a third frame number of images in the original image to obtain a corresponding photographed image; When the temperature of the electronic device is greater than or equal to a temperature threshold, one frame of the original image is processed using a single-frame algorithm to obtain a corresponding captured image; Saving the captured image; Among them, the first frame number is greater than the second frame number, the second frame number is greater than the third frame number, the time length for the electronic device to process image frames using the second camera algorithm is less than the time length for processing image frames using the first camera algorithm, and the time length for the electronic device to process image frames using the third camera algorithm is less than the time length for processing image frames using the second camera algorithm; the time length for the electronic device to process image frames using the single-frame algorithm is less than or equal to the time length for processing image frames using the third camera algorithm.

2. The image processing method according to claim 1, characterized in that: The method further comprises: When the first operation corresponds to the first photo-taking number, in response to the first operation, performing environmental detection on the scene to be photographed; Determine the number of processing frames and the processing algorithm corresponding to the first number of photographing times according to the environmental detection result; Using the processing algorithm corresponding to the first number of photographs, the image of the processing frame number corresponding to the first number of photographs is processed to obtain the corresponding captured image, and the processing frame number corresponding to the first number of photographs is the frame number of the original image collected by the first photograph.

3. The image processing method according to claim 2, characterized in that: The environmental detection includes at least illumination detection and dynamic range detection, and the environmental detection result includes at least illumination and dynamic range values; Determining the number of processing frames and the processing algorithm corresponding to the first number of photographing times according to the environmental detection result includes: When the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, determining the number of processing frames to be a1 frames, and determining the processing algorithm to be the A1 algorithm; When the illumination is less than the illumination threshold and the dynamic range value is less than the dynamic range threshold, determining the number of processing frames to be b1 frames, and determining the processing algorithm to be B1 algorithm; When the illumination is greater than or equal to the illumination threshold, and the dynamic range value is greater than or equal to the dynamic range threshold, determining the number of processing frames to be c1 frames, and determining the processing algorithm to be the C1 algorithm; When the illumination is greater than or equal to the illumination threshold and the dynamic range value is less than the dynamic range threshold, determining the number of processing frames to be d1 frames, and determining the processing algorithm to be the D1 algorithm; Wherein, a1, b1, c1 and d1 are all integers greater than or equal to 1.

4. The image processing method according to claim 3, characterized in that: Using the processing algorithm corresponding to the first number of photographing times, processing the image of the processing frame number corresponding to the first number of photographing times to obtain the corresponding photographed image, including: When the illumination is less than the illumination threshold and the dynamic range value is greater than or equal to the dynamic range threshold, the A1 algorithm is used to process the a1 frame image corresponding to the first photographing time to obtain a corresponding photographed image; When the illumination is less than the illumination threshold and the dynamic range value is less than the dynamic range threshold, the b1 frame image corresponding to the first photographing time is processed by the B1 algorithm to obtain a corresponding photographed image; When the illumination is greater than or equal to the illumination threshold, and the dynamic range value is greater than or equal to the dynamic range threshold, the C1 algorithm is used to process the c1 frame image corresponding to the first photographing time to obtain a corresponding photographed image; When the illumination is greater than or equal to the illumination threshold and the dynamic range value is less than the dynamic range threshold, the D1 algorithm is used to process the d1 frame image corresponding to the first photographing time to obtain a corresponding captured image.

5. The image processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine a frequency of a first operation on the first control, and a difference between a memory watermark and the memory usage; When the frequency meets the preset frequency condition or the difference is less than the preset difference, determining the number of photos to be taken in the background waiting; The processing algorithms corresponding to the number of times of taking photos in the background waiting are respectively used as the initial processing algorithms; The processing algorithm corresponding to the most recently determined number of photographing times is used as the target processing algorithm; Using the target processing algorithm to replace the initial processing algorithm of each of the number of photos waiting in the background; When any number of photographs waiting in the background is called for processing, the target processing algorithm is used to process the image of the processing frame number corresponding to the any number of photographs to obtain the corresponding captured image. The processing frame number corresponding to the any number of photographs is the number of frames in the original image captured by the any number of photographs.

6. The image processing method according to claim 5, characterized in that: The first interface refers to a photo taking interface, and the first control refers to a control for instructing photo taking.

7. The image processing method according to claim 5, characterized in that: The first interface refers to a video recording interface, and the first control refers to a control for instructing snapshot.

8. An electronic device, characterized in that: include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 7.

9. A chip, characterized in that: The chip is applied to an electronic device, and the chip comprises one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images