A photograph processing method and electronic device
By caching multiple frames and utilizing AI recognition technology, users can reselect key frames and adjust the resolution, solving the problem of capturing exciting moments and improving the shooting experience and photo quality.
Patent Information
- Application Number
- CN202410354553.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-25
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-03-25
AI Technical Summary
Users often struggle to capture memorable moments during shooting, leading to a diminished shooting experience.
Electronic devices use AI-powered image capture to identify and cache multiple frames. Users can select the best frames and generate high-quality photos, and can also reselect the best frames and adjust the resolution.
It improves the flexibility of users in selecting highlights and enhances photo quality, thus improving the shooting experience.
Smart Images

Figure CN120751272B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and more particularly to a photo processing method and electronic device. Background Technology
[0002] With the continuous advancement of terminal technology, users have increasingly higher demands for the ease of operation of electronic devices such as mobile phones and tablets; for example, users have increasingly higher requirements for the ease of taking photos with electronic devices. Currently, users may want to capture some exciting moments during the shooting process, such as a wonderful moment when an athlete is running.
[0003] During the shooting process, due to issues such as user reaction time delay and electronic device transmission delay, users often find it difficult to capture truly exciting moments, which greatly reduces the user's shooting experience. Summary of the Invention
[0004] This application provides a photo processing method and an electronic device that can fulfill the user's need to reselect the best frames.
[0005] The technical solution adopted in this application is as follows:
[0006] In a first aspect, a photo processing method is provided, the method comprising: in response to a shooting command, an electronic device saves a first photo and caches N frames of the first image; wherein the first photo includes a first highlight frame, the first highlight frame being identified by the electronic device using an AI capture function; the N frames of the first image are captured by the electronic device within a preset time period before and after the user inputs the shooting command; N≥3, and N is a positive integer.
[0007] In response to the user's first operation on the first photo, the electronic device displays N first thumbnails; each of the N first thumbnails corresponds one-to-one with a first image in N frames. In response to the user's selection of a first target thumbnail among the N first thumbnails, the electronic device saves a second photo; wherein the second photo includes a second highlight frame, which is the first image corresponding to the first target thumbnail selected by the user.
[0008] Based on the first aspect, the electronic device, while saving the first photo, also caches N frames of the first image. On this basis, when the user wants to reselect a highlight frame, the electronic device can respond to the user's first operation on the first photo by displaying N first thumbnails corresponding one-to-one with the N frames of the first image. Then, in response to the user's selection operation of a first target thumbnail among the N first thumbnails, the electronic device saves a second photo including a second highlight frame, which is the first image corresponding to the first target thumbnail selected by the user, thereby fulfilling the user's need to reselect a highlight frame.
[0009] In one possible implementation of the first aspect, the first and second photos are single photos; or, the first and second photos are the cover of a live photo series with a duration of x seconds, where the number of frames i ≥ 3, i is a positive integer, and x is a positive number.
[0010] In one possible implementation of the first aspect, in response to a user's selection operation of a first target thumbnail among N first thumbnails, the electronic device saves a second photo, including: in response to a user's selection operation of a first target thumbnail among N first thumbnails, the electronic device uses the first image corresponding to the first target thumbnail as a reference frame, merges M frames of first images with the reference frames to generate a second photo, and saves the second photo; M = N-1, and M is a positive integer.
[0011] Among them, the first image of frame M is the image in the first image of frame N excluding the reference frame.
[0012] In this embodiment, the electronic device uses the first image corresponding to the first target thumbnail as a reference frame, and merges the first image of M frames with the reference frame to generate a second photo, thereby solving the problem that the second photo generated after the user reselects a highlight frame has poor image quality.
[0013] In one possible implementation of the first aspect, the electronic device fuses the M-frame first image with a reference frame to generate a second photograph, including: the electronic device weighting the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the M-frame first image to generate the second photograph.
[0014] In one possible implementation of the first aspect, the electronic device performs weighted processing on the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the M-frame first image to generate a second photograph, including: the electronic device performs weighted processing on the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the M-frame first image to generate a first target image; the electronic device calls a preset image algorithm to process the first target image to generate a second photograph; wherein the image quality of the second photograph is higher than that of the first photograph.
[0015] In this embodiment, the electronic device first performs weighted processing on the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the first image in the M-frame first image to generate a first target image. Then, it calls a preset image algorithm to process the first target image to generate a second photo, thereby further improving the image quality of the second photo.
[0016] In one possible implementation of the first aspect, the electronic device caches a second image with a first resolution; the second photo corresponds to the second resolution; the method further includes: the electronic device receiving a second operation from a user on the second photo, the second operation being used to set the resolution of the second photo to a third resolution; the third resolution is less than the first resolution and greater than the second resolution.
[0017] In response to the second operation, the electronic device merges the second image at the first resolution and the second photograph at the second resolution, and saves the third photograph at the third resolution.
[0018] In this embodiment, since the electronic device also caches a second image at a first resolution, when the electronic device receives a user setting the resolution of the second photo to a third resolution, the electronic device can merge the second image at the first resolution and the second photo at the second resolution, and save the third photo at the third resolution, thereby fulfilling the user's need to adjust the resolution of the photo.
[0019] In one possible implementation of the first aspect, the electronic device merges a second image at a first resolution and a second photograph at a second resolution to save a third photograph at a third resolution, comprising: the electronic device downsampling the second image at the first resolution to generate a second image at a third resolution; the electronic device upsampling the second photograph at the second resolution to generate a second photograph at a third resolution; and the electronic device merging the second image at the third resolution with the second photograph at the third resolution to generate a third photograph at the third resolution.
[0020] Secondly, a photo processing method is provided, applied in an electronic device. The method includes: in response to a shooting command, the electronic device saves a first photo and caches N frames of the first image and a second image at a first resolution; wherein the first photo includes a first highlight frame, which is identified by the electronic device using an AI-based snapshot function; the N frames of the first image are captured by the electronic device within a preset time period before and after the user inputs the shooting command; N ≥ 3, and N is a positive integer.
[0021] In response to the user's first operation on the first photo, the electronic device displays N first thumbnails corresponding one-to-one with the N frames of the first image; in response to the user's selection operation on the first target thumbnail among the N first thumbnails, and the second operation on the first target thumbnail, the electronic device saves the second photo.
[0022] The first target thumbnail corresponds to the second resolution, and the second operation is used to set the resolution of the first target thumbnail to the third resolution, which is less than the first resolution and greater than the second resolution; the second photo corresponds to the third resolution, and the second photo includes the second highlight frame, which is the first image corresponding to the first target thumbnail selected by the user.
[0023] In this embodiment, when saving the first photo, the electronic device caches N frames of the first image and a second image at a first resolution. Then, in response to a user's first operation on the first photo, the electronic device can display N first thumbnails. Based on this, in response to a user's selection of a first target thumbnail among the N first thumbnails, and a second operation on the first target thumbnail, the electronic device can obtain a second photo at a third resolution, thereby simultaneously satisfying the user's needs to select highlights and adjust the resolution.
[0024] In one possible implementation of the second aspect, in response to a user's selection operation of a first target thumbnail among N first thumbnails and a second operation on the first target thumbnail, the electronic device saves a second photo, including: in response to a user's selection operation of a first target thumbnail among N first thumbnails and a second operation on the first target thumbnail, the electronic device uses a first image corresponding to the first target thumbnail as a reference frame, and fuses M frames of first images with the reference frames to generate a first target image; the first target image corresponds to a second resolution.
[0025] The electronic device fuses the first target image at a second resolution with the second image at a first resolution and saves the second photo at a third resolution; wherein, the first image of the M-frame is the image in the first image of the N-frame except for the reference frame.
[0026] In one possible implementation of the second aspect, the electronic device fuses the first target image at a second resolution with the second image at a first resolution to save the second photograph at a third resolution, including: the electronic device downsampling the second image at a first resolution to generate the second image at a third resolution; the electronic device upsampling the first target image at a second resolution to generate the first target image at a third resolution; and the electronic device fuses the first target image at a third resolution with the second image at a third resolution to save the second photograph at a third resolution.
[0027] Thirdly, a photo processing method is provided, applied in an electronic device. The method includes: in response to a shooting command, the electronic device saves a first photo and caches a second image at a first resolution; the first photo corresponds to the second resolution; in response to a second operation by a user on the first photo, the electronic device saves the second photo at a third resolution. The second operation is used to set the resolution of the first photo to the third resolution; the third resolution is greater than the second resolution and less than the first resolution.
[0028] In one possible implementation of the third aspect, in response to a second operation by the user on the first photograph, the electronic device saves a second photograph at a third resolution, including: in response to a second operation by the user on the first photograph, the electronic device merges the second image at the first resolution with the second photograph at the second resolution, and saves a third photograph at the third resolution.
[0029] In one possible implementation of the third aspect, the electronic device merges the second image at a first resolution with the second photograph at a second resolution to save the third photograph at a third resolution, including: the electronic device downsampling the second image at a first resolution to generate the second image at a third resolution; the electronic device upsampling the first photograph at a second resolution to generate the first photograph at a third resolution; and the electronic device merging the second image at a third resolution with the first photograph at a third resolution to save the second photograph at a third resolution.
[0030] Fourthly, an electronic device is provided, which has the functions described in any one of the first to third aspects above. These functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions.
[0031] Fifthly, an electronic device is provided, comprising: a memory and one or more processors; the memory stores computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method described in any one of the first to third aspects.
[0032] A sixth aspect provides a chip system comprising: at least one processor and an interface for receiving instructions and transmitting them to the at least one processor; the at least one processor executes instructions to cause an electronic device to perform the method described in any one of the first to third aspects.
[0033] A seventh aspect provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in any one of the first to third aspects.
[0034] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, enables the computer to perform the method described in any one of the first to third aspects.
[0035] The technical effects of any of the implementation methods in aspects four through eight can be referenced from the technical effects of different implementation methods in aspects one through three, and will not be elaborated here. Attached Figure Description
[0036] Figure 1 A schematic diagram of a viewfinder interface provided in an embodiment of this application;
[0037] Figure 2 This application provides a schematic diagram of an interface for enabling the AI capture function.
[0038] Figure 3 This is a schematic diagram of another interface for enabling the AI capture function provided in an embodiment of this application;
[0039] Figure 4 A schematic diagram illustrating an automatic identification of highlight frames provided in an embodiment of this application;
[0040] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;
[0041] Figure 6 A schematic diagram of the software framework of an electronic device provided in an embodiment of this application;
[0042] Figure 7 A schematic diagram of an interface for a photo processing method provided in an embodiment of this application;
[0043] Figure 8 A schematic flowchart illustrating a photo processing method provided in an embodiment of this application;
[0044] Figure 9 A schematic flowchart illustrating another photo processing method provided in an embodiment of this application;
[0045] Figure 10 A schematic diagram of an interface for another photo processing method provided in an embodiment of this application;
[0046] Figure 11 A schematic flowchart illustrating another photo processing method provided in an embodiment of this application;
[0047] Figure 12 A schematic flowchart illustrating another photo processing method provided in an embodiment of this application;
[0048] Figure 13 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation
[0049] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.
[0050] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" are used for descriptive purposes only and do not limit the quantity or execution order, nor should they be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Moreover, the terms "first" and "second" are not necessarily different. Therefore, a feature specified with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "at least one" means one or more, and "more than" means two or more.
[0051] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.
[0052] To facilitate understanding of the technical solutions provided in the embodiments of this application, the technical terms involved in this application will be explained below.
[0053] A "best moment" refers to a specific timeframe in which the subject's state and / or movement are at their optimal. "State" and "movement" are defined for both still life and non-still life shooting scenarios, respectively. In still life shooting scenarios, a best moment occurs when the subject is in a relatively static state within the physical space, and the still object is clearly imaged in the camera's view, without shaking or noise. In non-still life shooting scenarios, a best moment occurs when the subject's movement is at its best within the physical space, and the non-still object is clearly imaged in the camera's view, without shaking or noise.
[0054] For example, when filming famous historical sites, the moment when the building is completely, clearly, and without shaking or noise in the camera's image is considered a highlight. When filming a hurdler, the moment when the athlete jumps and clears the hurdle with a clear, shaking, and noise-free image is considered a highlight. When filming a close-up portrait, the moment when the person is smiling with their eyes open with a clear, shaking, and noise-free image is considered a highlight.
[0055] Among them, the image frames captured by the camera at the exciting moment are called exciting frames, and the photos output after image processing of the exciting frames are called exciting photos.
[0056] It should be noted that a highlight frame can also be called a highlight frame image or a highlight image frame, etc., and the terms "highlight frame" and "highlight frame image" are merely descriptive examples. The description of highlight frames in this application embodiment does not impose any limitations. For example, the above-mentioned highlight frame can also be called an excellent frame image, a preset frame image, a keyframe, etc.
[0057] Live Photos: These record and dynamically display image and sound information within a preset time period (e.g., 1.5 seconds) before and after a user takes a photo. They are available in media formats that include both still images and dynamic video clips. A Live Photo can contain multiple image frames. After the Live Photo is taken, the electronic device automatically selects one image frame according to preset rules (e.g., selecting the image frame with the highest clarity among multiple image frames) and sets this image frame, after image processing, as the default cover for the Live Photo. Of course, users can also select other image frames from the Live Photo as the cover photo, but their clarity will be lower than the default cover. Correspondingly, users can perform the same editing operations on Live Photos as on regular photos.
[0058] Image frame: The smallest unit that makes up a video (or a live photo, a single photograph, etc.). A video can include several image frames, and a live photo can also include several image frames. The difference between a video and a live photo is that a video is longer than a live photo. A single photograph consists of one image frame.
[0059] Pixel value: This is the value assigned by the computer when the original image is digitized. It represents the average brightness information of a small square in the original image, or the average reflectance (transmission) density information of that small square.
[0060] The scenario for this application is described below:
[0061] The solution in this application embodiment can be applied to scenarios involving capturing exciting frames. For example, both manual and automatic capture methods can be used to capture exciting frames. The manual capture method refers to the user simultaneously taking photos or videos manually to obtain the desired image (i.e., manually capturing exciting frames).
[0062] Automatic snapshot mode refers to the ability of electronic devices to capture exciting frames using artificial intelligence (AI) snapshot functions. For example, AI snapshot function is an intelligent snapshot feature provided by a camera application, which differs from manual snapshot mode and can achieve snapshots without user intervention. AI snapshot function can capture exciting frames, such as frames of successful basketball shots, frames of reaching the highest point of a jump, frames of completing a look-back action, and frames of specific subjects, including rainbows / sunsets, etc. This application does not specifically limit this approach.
[0063] The above-mentioned automatic capture method can be applied to scenarios where camera applications are used in electronic devices such as mobile phones and tablets (the following embodiments take mobile phones as an example), and in such scenarios, capture wonderful frames.
[0064] For example, when capturing exciting frames using automatic capture, the images can be displayed on the phone. Figure 1 During the process of framing the view 101 shown (i.e., before taking a photo or before recording a video), the best frames are identified from the preview image and stored.
[0065] For example, when capturing exciting frames using automatic capture, the image can be displayed on the phone. Figure 1 During the process of framing the view 102 shown (i.e., recording video), the best frames are identified from the preview image and stored.
[0066] The following describes the process of automatic snapshot capture using a mobile phone:
[0067] In some embodiments, the camera application settings interface of the mobile phone includes a control a. The mobile phone can receive user operations a on control a (such as clicking, long pressing, swiping, etc., with clicking as an example below), and in response to user operations a on control a, the mobile phone can enable the AI capture function.
[0068] For example, a mobile phone can display Figure 2 The viewfinder interface 201 shown is for pre-capture shooting. Interface 201 includes a settings button 202. In response to the user clicking the settings button 202, the phone can display... Figure 2 The interface 203 shown is the camera application settings interface, which includes a "Smart Photography" option. The phone responds to the user's click on the "Smart Photography" option in interface 203 by displaying... Figure 2The interface shown is 204. Interface 204 is the settings interface for "Smart Photography" (also part of the camera application's settings interface). Interface 204 includes an "Auto Capture" setting 205, which contains a switch 206. The switch 206 in interface 204 is in the off state; it is control a. In response to the user's click on switch 206 in interface 204, the phone can activate the AI capture function. For example, in response to the user's click on switch 206 in interface 204, the phone can display... Figure 2 The interface 207 shown also includes a switch 206, and the state of switch 206 in interface 207 is the on state.
[0069] In this embodiment, after the AI capture function is enabled, in response to a preset event that triggers the display of the viewfinder, the mobile phone can start AI capture to capture exciting frames.
[0070] For example, after enabling the AI capture function, the electronic device can display... Figure 2 The interface shown is 207. In response to the user's return operation on interface 207, the phone can display... Figure 2 The interface 203 shown is as shown. In response to the user's return operation on interface 203, the phone can display... Figure 2 The viewfinder interface 201 is shown. At this point, the phone begins AI-powered image capture. That is, the preset event is when the user performs a return operation on interface 203.
[0071] For example, after enabling the AI capture function, in response to the user's click on the camera app icon on the phone's home screen, the phone can display the viewfinder interface before taking the photo (e.g., ...). Figure 1 The viewfinder interface shown is 101. At this point, the phone begins AI-powered snapshot capture. That is, the preset event is the user clicking the camera app icon on the phone's home screen.
[0072] In other embodiments, the viewfinder of the phone's camera application includes a preset control b for enabling the AI snapshot function. The phone can receive user operations b on the control b (such as clicking, long-pressing, swiping, etc., with clicking as an example below). In response to the user's operation b on the control b, the phone can enable the AI snapshot function and simultaneously begin AI snapshot capture to capture exciting frames.
[0073] Control b is Figure 3 Taking icon 302 in the viewfinder 301 as an example, the phone can display... Figure 3 The viewfinder interface 301 shown is for taking a photo before shooting. Interface 301 includes an icon 302. The phone can receive the user's click operation on icon 302. In response to the user's click operation on icon 302, the phone can activate the AI capture function and start AI capture simultaneously.
[0074] refer to Figure 2 and Figure 3 Users can operate the AI capture function controls (such as controls a and b) through the same path, and then turn off the AI capture function through the same controls. The process is the same as described in the previous embodiment and will not be repeated here. By implementing the above method, users can turn the AI capture function on / off at any time according to their needs. In this way, when users need to automatically capture wonderful photos of exciting moments, the phone can provide the above service; when users do not need to automatically capture, the phone can turn off the AI capture function, saving computing costs, reducing power consumption, and improving the user experience.
[0075] In some embodiments, while the phone is displaying the viewfinder, the phone's camera application continuously requests new images from the camera. Correspondingly, the camera continuously captures images and reports them to the phone's camera application so that the camera application can display the images in the viewfinder. It should be understood that the images displayed by the phone's camera application in the viewfinder can be called preview images.
[0076] For example, the mobile phone can identify each frame of the preview image captured in real time during the preview process and determine whether the frame is a highlight frame. After identifying a highlight frame, the mobile phone can automatically perform image processing on the highlight frame and save the highlight photo in the mobile phone's album application (or gallery application). It should be understood that in this embodiment, the highlight photo output by the mobile phone is a single photo.
[0077] For example, during the preview process, the phone receives a shooting command from the user. In response to the user's shooting command, the phone acquires multiple preview images within a preset time period before and after the user inputs the shooting command. For instance, if the preset time period is 1.5 seconds, after receiving the user's shooting command, the phone acquires multiple preview images within 1.5 seconds before and after the time node "receiving the user's shooting command". Then, for each of these multiple preview images, the phone can identify whether it includes a highlight frame. After identifying the highlight frame, the phone generates a live photo based on the multiple preview images, performs image processing on the highlight frame, sets the processed highlight frame as the default cover of the live photo, and then saves the live photo in the phone's photo album application (or gallery application).
[0078] The live photo saved on the phone includes a single JPG frame and a short video clip. The JPG frame serves as the default cover image for the live photo. In practice, the image quality (such as sharpness and resolution) of the JPG frame is higher than that of the video clip. This is because when generating a live photo, the phone uses an image enhancement algorithm on the selected frame to produce the higher-quality JPG frame. Other frames undergo simpler processing before being edited into a video, thus reducing computational effort and device power consumption.
[0079] In some embodiments, automatic capture allows the phone to automatically identify and capture memorable moments without requiring user intervention, simplifying the user experience and preventing missed opportunities. Furthermore, to enhance the user experience and avoid generating excessive and redundant photos, the phone can identify detailed shooting scenes and set a limit on the number of memorable photos for each scene. When the number of memorable photos for a scene reaches its limit, the phone will stop automatically capturing photos of that scene. Alternatively, the phone can set an automatic capture limit; once the limit is reached, the phone will stop automatically capturing photos.
[0080] The following describes the process of a mobile phone recognizing highlights:
[0081] As an example of this application, a preset algorithm for automatic image capture is used to identify exciting frames. For example, the preset algorithm for automatic image capture is used to automatically take pictures when exciting moments such as looking back, jumping, running, and cats and dogs are detected.
[0082] For example, such as Figure 4 As shown, a mobile phone captures an athlete performing a long jump. With automatic capture enabled, the phone's viewfinder displays the athlete's movement at different moments during the jump. For example, the phone can sequentially display the first preview image, the second preview image, the third preview image, ..., the kth preview image. Simultaneously, using the preset algorithm corresponding to automatic capture, the phone can identify each of these preview images and determine the highlight frames.
[0083] For example, in the long jump, the flat position after takeoff can be considered the optimal state for completing the long jump, i.e. Figure 4 The state shown in the 3rd frame preview image. After identifying the 1st to kth frame preview images, the phone can determine that the 3rd frame preview image is a highlight frame.
[0084] In this embodiment, the preset algorithm corresponding to automatic image capture can define the optimal state for different scenarios. For example, in long jump, the flat-shaped state after the person takes off can be called the optimal state for completing the long jump. Similarly, in jumping, the state at the highest point after the person leaves the ground can be called the optimal state for jumping, and so on, without limitation.
[0085] As another example of this application, a preset model is used to automatically capture images using a mobile phone to identify highlights. For example, the preset model for automatically capturing images is used to output an aesthetic score for each preview image frame; wherein the aesthetic score indicates the highlight level of the preview image.
[0086] Taking the capture of a person's actions with a mobile phone as an example, the system automatically captures a preset model to obtain the person's actions, such as jumping, looking back, throwing objects, playing ping-pong, playing badminton, running, splashing water, playing baseball, playing rugby, etc. Based on the optimal completion point of each action, the system determines the completion level of the action in any preview frame and outputs an aesthetic score for that preview image. The higher the completion level of the action, the higher the aesthetic score. After recognizing each preview frame, the mobile phone can identify the frame with the highest aesthetic score as the "highlight frame."
[0087] Through the aforementioned automatic capture method, the mobile phone can automatically identify key frames and take excellent photos. However, if the key photos taken by the mobile phone do not meet the user's needs—that is, the key photos automatically taken by the mobile phone are not the key photos the user wants—it reduces the user's shooting experience. Based on this, embodiments of this application provide a photo processing method that allows the mobile phone to reselect key frames after automatically taking key photos and generate key photos based on the reselected key frames.
[0088] For example, this method can be applied to smartphones, personal digital assistant (PDA) computers, tablet computers, laptop computers, camcorders, video recorders, cameras, smartwatches, smart wristbands, cellular phones, in-vehicle computers, and other electronic devices with camera applications. This application does not impose any special limitations on the specific form of the electronic device.
[0089] refer to Figure 5 This is a hardware structure diagram of an electronic device provided in an embodiment of this application. Figure 5As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a positioning module 181, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.
[0090] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0091] Processor 110 may include one or more processing units, such as a central processing unit (CPU), application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.
[0092] In some embodiments, the processor 110 can perform automatic capture processing, as well as reselect the best frames and regenerate the best photos.
[0093] The charging management module 140 receives charging input from a charger, which can be either a wireless or wired charger. The power management module 141 connects to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.
[0094] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.
[0095] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include one or more filters, switches, power amplifiers, low-noise amplifiers (LANs), etc. The mobile communication module 150 can receive electromagnetic waves via the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functions of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.
[0096] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating one or more communication processing modules. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.
[0097] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0098] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.
[0099] Electronic device 100 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 194, and application processor. The ISP processes data fed back from the camera 193. For example, when taking a picture, the shutter is opened, light is transmitted through the lens to the camera's photosensitive element, the light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into a visible image. The ISP can also perform algorithmic optimization of image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature. In some embodiments, the ISP can be integrated into the camera 193.
[0100] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.
[0101] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.
[0102] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 110 by running the instructions stored in internal memory 121. For example, processor 110 can display different content on display screen 194 in response to an operation to unfold display screen 194 by executing instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phone book, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.
[0103] Electronic device 100 can implement audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, such as music playback and recording.
[0104] Buttons 190 include a power button and volume buttons. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with electronic device 100. Electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.
[0105] The software system of the aforementioned electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture as an example to exemplify the software structure of the electronic device 100.
[0106] refer to Figure 6 This is a software structure block diagram of an electronic device 100 provided in an embodiment of this application. For example... Figure 6As shown, a layered architecture can divide software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the layered architecture can be divided into four layers, from top to bottom: the application layer (application layer), the application framework layer (framework layer), the hardware abstraction layer (HAL), and the kernel layer (also known as the driver layer).
[0107] The application layer can include a series of application packages. These application packages can be applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.
[0108] In the embodiments of this application, such as Figure 6 As shown, this application layer can include the system camera application (also known as the camera app). The camera app can provide functions such as taking photos, recording videos, and AI-powered snapshots.
[0109] Camera apps can be used in the viewfinder (such as...) Figure 1 The preview image reported from the underlying layer is displayed in viewfinder interfaces 101 and 102. The electronic device may include one or more cameras, each of which can be used to capture the preview image.
[0110] The framework layer provides the application programming interface (API) and programming framework to the application layer. The framework layer includes some predefined functions. For example... Figure 6 As shown, the frame layer can provide camera services.
[0111] The camera service can be used to transmit requests from the application layer to lower layers (such as the HAL layer), such as requests to end shooting, requests to switch cameras, requests to adjust focus, etc. The camera service can also be used to process the preview frame data reported by the lower layer (such as the HAL layer) and then upload it to the upper layer application.
[0112] The HAL layer is an abstraction layer located between the kernel layer and the Android runtime. The HAL layer can be a wrapper for hardware drivers, providing a unified interface for calls from upper-layer applications.
[0113] like Figure 6As shown, the HAL layer includes the camera hardware abstraction layer (camera HAL). The camera HAL provides a standard Hardware Abstraction Layer Interface Definition Language (HAL interface definition language) for calls to upper layers (such as the camera service), maintaining normal communication with them. Furthermore, the camera HAL uses standard HAL interfaces, such as HAL 3.0, to control the kernel layer, obtain preview frame data reported by the kernel layer, and then report it to the upper layers (such as the camera service).
[0114] The camera HAL is the core software framework of the camera device. The camera HAL can include various image processing algorithms, such as face recognition algorithms, skin smoothing algorithms, age recognition algorithms, fill light algorithms, sharpening and high-definition processing algorithms, background blurring algorithms, high dynamic range image algorithms, multi-frame fusion processing algorithms, etc., without limitation.
[0115] The kernel layer includes a camera driver, which is used to start the camera device and transmit image data acquired by the camera device to higher layers (such as the HAL layer). The camera device may include one or more cameras, each including a camera lens and an image sensor. The ISP described above can be configured separately from the camera device (e.g., a camera). In some embodiments, the ISP can be integrated into the camera device (e.g., a camera).
[0116] The ISP (Image Signal Processor) and camera device are the main equipment for capturing video or images. Light signals reflected from the scene are reflected by the camera lens onto the image sensor and converted into electrical signals. These electrical signals are processed by the ISP and can be used as a raw parameter stream, transmitted to the upper layer by the camera driver. Furthermore, the camera driver can also receive notifications from the upper layer (such as instructions to turn the camera on or off), and based on these notifications, send a functional processing parameter stream to the camera device to turn the corresponding camera on or off.
[0117] The photo processing method provided in this application embodiment can be applied to electronic devices having the above-described hardware and software structures.
[0118] Specifically, the photo processing method provided in this application includes the following three embodiments: Embodiment 1: Selecting a highlight frame. Embodiment 2: Adjusting the resolution. Embodiment 3: Selecting a highlight frame and adjusting the resolution. The following description uses a mobile phone as an example to illustrate each of the three embodiments.
[0119] Example 1: Selecting a Highlight Frame.
[0120] In some embodiments, the mobile phone displays a viewfinder (such as viewfinder 101 and viewfinder 102 described above), which can be used to display preview images captured by the mobile phone. For example, in response to a shooting command, the mobile phone saves a first photo and caches N frames of the first image. The first photo includes a first highlight frame, which is identified by the mobile phone using the AI capture function described above; the N frames of the first image are captured by the mobile phone within a preset time period before and after the user inputs the shooting command; N ≥ 3, and N is a positive integer.
[0121] It should be noted that the specific implementation method of automatically identifying the first exciting frame using the AI capture function of the mobile phone can be referred to the description in the above embodiments, and will not be repeated here.
[0122] For example, the first photo can be stored in the access area of the phone's photo album application (or gallery application). For instance, N frames of the first image can be cached in the phone's internal memory 121. When the internal memory 121 includes read-only memory (ROM), the N frames of the first image can be cached in the memory space of the ROM.
[0123] Optionally, the N frames of the first image cached by the phone can be the original RAW domain image. That is, after the phone captures N frames of the first image within a preset time period before and after the user inputs the shooting command, the N frames of the first image are directly cached in the phone's storage space.
[0124] Alternatively, the first N frames cached by the phone can be RGB domain images or YUV domain images. That is, after the phone captures N RAW domain images within a preset time period before and after the user inputs the shooting command, it converts the N RAW domain images into RGB domain images or YUV domain images to obtain the first N frames, and caches the first N frames in the phone's storage space.
[0125] Optionally, the phone can compress the first N frames of the image and cache them in the phone's storage space, thereby avoiding a large amount of image data occupying the phone's memory space.
[0126] In some embodiments, after the phone saves the first photo to the photo album application, the phone can, in response to the user launching the photo album application, view the saved first photo; or, edit the first photo, such as cropping, rotating, saving, sending, etc., without limitation. For example, the phone can also, in response to the user reselecting a frame from the first photo in the photo album application. For example, the phone displays... Figure 7 The interface 401 shown includes a first photo 402.
[0127] For example, interface 401 also includes an edit button 403, and in response to a user clicking the edit button 403, the phone displays... Figure 7 The interface 404 shown includes a "Select Highlight Frame" option 405. In response to the user's click on the "Select Highlight Frame" option 405, the phone displays... Figure 7 The interface 406 shown includes N first thumbnails. Each of these N first thumbnails corresponds one-to-one with N frames of first images cached on the phone. For example, the N frames of first images cached on the phone include: frame 1, frame 2, frame 3, ..., frame N; the N first thumbnails displayed on interface 406 include: thumbnails... Figure 1 abbreviation Figure 2 abbreviation Figure 3 ..., thumbnail N. Therefore, the first image of the first frame mentioned above and the thumbnail... Figure 1 Correspondingly, the first image in the second frame and its thumbnail Figure 2 Correspondingly, the first image of frame 3 and its thumbnail Figure 3 Correspondingly, ... the first image in frame N corresponds to thumbnail N.
[0128] It should be noted that the above-mentioned N first thumbnails correspond one-to-one with the N frames of first images cached by the mobile phone means that the image content of the first thumbnail is the same as that of the corresponding first image.
[0129] For example, such as Figure 7 The interface 406 shown responds to the user's swiping operation (such as swiping left or right) on the first thumbnail. The phone can scroll and display each of the N first thumbnails. For example, the phone scrolls and displays the aforementioned thumbnails. Figure 1 abbreviation Figure 2 abbreviation Figure 3 ..., N thumbnails. Based on this, in response to the user's selection of thumbnails from the N first thumbnails... Figure 1 The selection operation (or the first target thumbnail) is displayed on the phone. Figure 7 The interface shown is 407, and the abbreviated version of interface 407 is shown below. Figure 1 The image is selected. Then, in response to the user's confirmation, the phone generates a second photo. This second photo includes a second highlight frame, which is the thumbnail selected by the user. Figure 1 The corresponding first image.
[0130] Optionally, the first and second photos can be single images. The first photo is generated by the phone based on the first highlight frame; that is, the phone captures the first highlight frame using AI capture technology and then generates the first photo. Similarly, the second photo is generated by the phone based on the second highlight frame, i.e., the image is generated when the user selects the aforementioned thumbnail. Figure 1 After the user selects the second best frame, the phone generates a second photo.
[0131] Optionally, the first and second photos are the cover images of a live photo lasting x seconds, where the number of frames i ≥ 3, i is a positive integer, and x is a positive number. Based on this, the first photo is the cover image of the live photo generated by the phone based on the first highlight frame; that is, after the phone captures the first highlight frame using AI capture technology, the phone processes the first highlight frame and sets it as the cover image of the live photo. Correspondingly, the second photo is the cover image of the live photo generated by the phone based on the second highlight frame; that is, the cover image is the one the user selects when choosing the above thumbnail. Figure 1 After the corresponding first image (i.e. the second highlight frame selected by the user), the phone processes the second highlight frame and sets it as the cover of the live photo.
[0132] In summary, by adopting the solution of this application embodiment, when the mobile phone receives a shooting command input by the user, it can capture N first images within a preset time period before and after the user inputs the shooting command, and cache the N first images in the mobile phone. Then, when the user wants to select a highlight frame, the mobile phone can display N first thumbnails corresponding one-to-one with the N first images for the user to choose from, and generate a second photo based on the selected highlight frame. In this way, the user can obtain the truly memorable moment captured, satisfying the user's need to select a highlight frame and improving the user experience.
[0133] In some embodiments, such as Figure 8 As shown, when the phone saves the first photo (e.g., a JPG format photo), it also caches N frames of the first image. Then, in response to the user's editing operation on the first photo (e.g., selecting a highlight frame), the phone processes the N frames of the first image through the ISP, generating N first thumbnails. Next, in response to the user's selection of a first target thumbnail, the phone uses the first image corresponding to the first target thumbnail as a reference frame, and merges M frames of the first image with the reference frame to generate the second photo. Here, the M frames of the first image are all the images in the N frames of the first image except for the reference frame, M = N-1, and M is a positive integer.
[0134] For example, in conjunction with the above Figure 7 As shown, if the user selects a thumbnail as the first target thumbnail... Figure 1 The phone will then abbreviate. Figure 1 The corresponding first image (i.e., the first image of frame 1) is used as the reference image. The first images of frames M (i.e., the first images of frames 2, 3, ..., and the first images of frames N) are fused with the reference frame (the first image of frame 1) to generate the second photo.
[0135] In some embodiments, such as Figure 6As shown, after the application layer's camera application receives the user's editing operation (such as selecting a highlight frame), it sends the user's operation command (i.e., the command to select a highlight frame) to the HAL layer's camera HAL. Upon receiving the operation command, the HAL layer's camera HAL reads the N frames of the first image cached by the phone, processes them, and generates N first thumbnails. Then, the HAL layer's camera HAL reports these N first thumbnails to the application layer's camera application, enabling the camera application to display them.
[0136] Furthermore, the camera application at the application layer receives a thumbnail of the first target (such as a thumbnail). Figure 1 After the user selects a thumbnail, the corresponding message (i.e., the user's selected thumbnail) will be sent. Figure 1 The message is sent to the camera HAL in the HAL layer. After receiving the corresponding message, the camera HAL will display the thumbnail selected by the user. Figure 1 The corresponding first image is used as a reference frame, and then the first image of the M-frame is fused with the reference frame to generate the second image.
[0137] Optionally, the camera HAL in the HAL layer fuses the first image of frame M with the reference frame to generate the first target image. Then, the camera HAL calls the preset image processing algorithm corresponding to the HAL layer (such as the various image processing algorithms mentioned above) to process the first target image and generate the second photo.
[0138] For example, the preset image processing algorithm may include noise reduction (NR) processing, which can remove noise points from the image, eliminate blur bands, enhance color saturation, and improve realism. Therefore, the image quality of the second photo generated after noise reduction processing is higher than that of the first photo.
[0139] In some embodiments, the fusion of the M-frame first image with the reference frame refers to: the mobile phone acquiring the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the M-frame first image. Then, the mobile phone performs weighted processing on the pixel values of each pixel in the reference frame and the pixel values of each pixel in each frame of the M-frame first image.
[0140] For example, such as Figure 9 As shown, assuming the reference frame is the first frame (first image), and the M frames (first images) include the second frame (first image), the third frame (first image), ..., the Nth frame (first image), the mobile phone performs weighted processing on the pixel value of each pixel in the first frame (first image) and the pixel values of each pixel in the first frame (first image), the third frame (first image), ..., the Nth frame (first image) to obtain the second image.
[0141] In summary, by employing the above method, in response to the user's selection of a highlight frame, the phone uses the selected highlight frame (such as the first image corresponding to the first target thumbnail mentioned above) as a reference frame and merges it with other first images (i.e., the first image of frame M) to obtain the second photo, thus solving the problem of poor image quality in reselected highlight frames. Furthermore, since this second photo is processed by the phone using a preset image processing algorithm, its image quality is higher than that of the first photo. This not only satisfies the user's need to select a highlight frame but also further improves the image quality, thereby enhancing the user experience.
[0142] Example 2: Adjusting the resolution.
[0143] In some embodiments, when the phone saves the first photo, it also caches a second image at a first resolution. The first photo corresponds to the second resolution, and the first resolution is greater than the second resolution.
[0144] For an example of how the phone saves the first photo, please refer to the description in the above embodiments, which will not be repeated here.
[0145] It should be noted that in related technologies, the pixel arrangement of image sensors is generally a Bayer array. In a Bayer array, each pixel unit consists of one red pixel (R), two green pixels (G), and one blue pixel (B). Each pixel comprises four sub-pixels, arranged in a 2×2 matrix. This arrangement results in a higher pixel count and thus a higher original resolution for the image sensor.
[0146] However, due to the different image output modes of image sensors, even when images are output from the aforementioned image sensors, the resolution of the output image may be lower than the original resolution of the image sensor. For example, the original resolution of an image sensor may be 50MP (megapixels), while a mobile phone's camera application defaults to taking photos at 12MP (megapixels), meaning that the resolution of photos taken by the mobile phone is 12MP.
[0147] In summary, it is understandable that image sensors can output high-resolution images, but because mobile phone camera applications have a lower default resolution, the resolution of photos taken by the phone is lower than the original resolution of the image sensor. Therefore, embodiments of this application can also adjust the resolution of a photo in response to user input after the phone has taken the photo.
[0148] For example, the phone displays Figure 10The interface 501 shown includes a first photo 502. Interface 501 also includes an edit button 503; in response to a user clicking the edit button 503, the phone displays... Figure 10 The interface 504 shown includes an "Adjust resolution" option 505. In response to the user's click on the "Adjust resolution" option 505, the phone displays... Figure 10 The interface 506 shown includes a "resolution input box" 507. The user can input the desired resolution in the "resolution input box" 507. For example, in response to the user inputting a third resolution (e.g., 30MP) in the "resolution input box" 507, the phone displays... Figure 10 The interface 508 shown includes a second photo 509 with a resolution of 30MP.
[0149] It should be noted that, Figure 10 Taking a resolution value that the user can input as an example, the mobile phone can also display multiple resolution options for the user to choose from, subject to the actual setting. This application embodiment does not limit this.
[0150] Optional, such as Figure 11 As shown, when the phone saves the first photo (e.g., a JPG format photo), it also caches a second image at the first resolution. Then, in response to the user's editing operation on the first photo (e.g., adjusting the resolution), the phone reads the first image at the first resolution, merges the second image at the first resolution with the first photo at the second resolution, and processes the merged image using a preset image algorithm to generate a second photo at the third resolution.
[0151] Optionally, assume the first resolution is 50MP, the second resolution is 12MP, and the user-adjusted resolution (i.e., the third resolution) is 30MP. Based on this, the phone downsamples the second image at the first resolution to generate the second image at the third resolution, that is, downsampling the resolution of the second image to 30MP. The phone then upsamples the first image at the second resolution to generate the first image at the third resolution, that is, upsampling the resolution of the first image to 30MP.
[0152] Then, the phone merges the second image at the third resolution with the first photo at the third resolution to generate the second photo at the third resolution.
[0153] For example, after downsampling the second image at the first resolution, the mobile phone generates a second image at the third resolution, i.e., a 30MP second image. Then, the mobile phone filters the 30MP second image to obtain high-frequency and low-frequency images. The high-frequency image refers to areas in the second image with large changes in brightness (or grayscale), while the low-frequency image refers to areas in the second image with small changes in brightness (or grayscale).
[0154] For example, a mobile phone can merge a 30MP high-frequency image with a 30MP first photo to generate a 30MP second photo. This results in a second photo with richer details and higher image quality.
[0155] In summary, by adopting the solution of this application embodiment, the mobile phone can provide the user with an interface for adjusting the resolution. In response to the user's operation of adjusting the resolution, the mobile phone can generate a second photo with a third resolution based on the current photo (i.e., the first photo) and the cached second image. This not only improves the resolution of the photo and makes the photo clearer, but also improves the user experience.
[0156] Example 3: Select a highlight frame and adjust the resolution.
[0157] In some embodiments, in response to a user's selection operation, after the mobile phone selects a highlight frame, a second photo is generated in accordance with the method described in Embodiment 1, and the second photo corresponds to a second resolution. Based on this, in response to a second user operation on the second photo (i.e., setting the resolution of the first photo to a third resolution), the mobile phone generates a third photo at a third resolution based on the second image at the first resolution and the second photo at the second resolution.
[0158] It is understandable that the second image at the first resolution is cached by the mobile phone after receiving the shooting command. For an example of the second image at the first resolution, please refer to the description in Embodiment 2 above; it will not be repeated here. Similarly, for examples of the first, second, and third resolutions, please refer to the description in Embodiment 2 above; they will not be repeated here.
[0159] For example, in response to a user's second operation on the second photo, the mobile phone merges the second image at the first resolution and the second photo at the second resolution to generate a third photo at the third resolution.
[0160] For example, the phone downsamples a second image at a first resolution to generate a second image at a third resolution. The phone then upsamples the second image at the second resolution to generate a second image at the third resolution. Finally, the phone merges the second image at the third resolution with the second image at the third resolution to generate a third image at the third resolution.
[0161] Taking a first resolution of 50MP, a second resolution of 12MP, and a third resolution of 30MP as an example, the phone downsamples the 50MP second image to 30MP and upsamples the 12MP second photo to 30MP. Then, the phone merges the 30MP second image with the 30MP second photo to generate a 30MP third photo.
[0162] The specific implementation process of the mobile phone fusing the 30MP second image with the 30MP second photo can be referred to the description in Embodiment 2 above, and will not be repeated here.
[0163] In other embodiments, after the phone saves the first photo, it caches N frames of the first image and a second image at a first resolution. Based on this, in response to a user's first operation on the first photo, the phone displays N first thumbnails; in response to a user's selection of a first target thumbnail among the N first thumbnails, and a second operation on the first target thumbnail, the phone generates a second photo. That is, in this embodiment, the user can simultaneously select key frames and adjust the resolution.
[0164] The first target thumbnail corresponds to the second resolution, and the second photo corresponds to the third resolution.
[0165] For example, such as Figure 12 As shown, after the phone receives a first photo (e.g., a JPG format photo), it caches N frames of the first image and a second image at a first resolution. Then, in response to a user's editing operation on the first photo (e.g., selecting a highlight frame), the phone processes the N frames of the first image via the ISP to generate N first thumbnails. Furthermore, in response to a user's selection of a first target thumbnail and a second operation on the first target thumbnail, the phone uses the first image corresponding to the first target thumbnail as a reference frame and merges M frames of the first image with the reference frame to generate the first target image.
[0166] It is understood that in this embodiment, the first target thumbnail corresponds to the second resolution, and the second operation is used to set the resolution of the first target thumbnail to the third resolution. The mobile phone uses the first image corresponding to the first target thumbnail as a reference frame, then fuses the M-frame first image with the reference frame to generate a first target image with the second resolution.
[0167] Then, the phone merges the first target image at the second resolution with the second image at the first resolution to generate a second photo at the third resolution.
[0168] For example, the mobile phone downsamples the second image at a first resolution to generate a second image at a third resolution; the mobile phone upsamples the first target image at a second resolution to generate a first target image at a third resolution. Then, the mobile phone merges the first target image at the third resolution with the second image at the third resolution to generate a second photo at the third resolution.
[0169] The specific implementation methods for the mobile phone to generate the first target image and the second photo with the third resolution can be referred to the examples in Embodiment 1 and Embodiment 2 above, and will not be repeated here.
[0170] In summary, using the solution of this application embodiment, the mobile phone can generate a second photo in response to the user's selection of a highlight frame. After generating the second photo, it can generate a third photo with a third resolution in response to the user's resolution adjustment operation. Alternatively, the mobile phone can also generate a second photo with a third resolution in response to both the user's selection of a highlight frame and the user's resolution adjustment operation. In other words, the mobile phone can provide users with multiple implementation methods, simultaneously satisfying the user's needs for selecting highlight frames and adjusting resolution, greatly improving the user experience.
[0171] It should be noted that the contents described in the various embodiments of this application can explain the technical solutions in other embodiments of this application, and the technical features described in the various embodiments can also be applied in other embodiments. New solutions can be formed by combining the technical features in other embodiments. This application only provides an exemplary list of several embodiments for illustration and does not mean that this application is limited thereto.
[0172] This application provides an electronic device that may include a camera, a memory, and one or more processors. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the various functions or steps described in the above embodiments. The structure of the electronic device can be referred to above. Figure 5 The structure of the electronic device 100 shown.
[0173] This application also provides a chip system for use in electronic devices. For example... Figure 13 As shown, the chip system 1100 includes at least one processor 1101 and at least one interface circuit 1102. The processor 1101 can be one of the types described in the above embodiments. Figure 5 The processor 110 shown can be used as an interface circuit 1102, for example, as an interface circuit between the processor 1101 and external memory; or as an interface circuit between the processor 1101 and internal memory.
[0174] The processor 1101 and interface circuit 1102 described above can be interconnected via a line. For example, interface circuit 1102 can be used to receive signals from other devices (e.g., the memory of electronic device 100). As another example, interface circuit 1102 can be used to send signals to other devices (e.g., processor 1101). Exemplarily, interface circuit 1102 can read instructions stored in memory and send those instructions to processor 1101. When the instructions are executed by processor 1101, the electronic device can perform the various functions or steps performed by the mobile phone in the above embodiments. Of course, the chip system may also include other discrete components, and this application embodiment does not specifically limit this.
[0175] This application also provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.
[0176] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.
[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules according to the system, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0178] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0179] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.
[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0182] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A photo processing method, characterized in that, Applied in electronic devices; the method includes: In response to a shooting command, the electronic device saves a first photo and caches N frames of the first image; wherein, the first photo is a photo generated based on a first highlight frame, and the first highlight frame is a frame identified by the electronic device from the N frames of the first image using AI capture function; the N frames of the first image are captured by the electronic device within a preset time period before and after the user inputs the shooting command; N≥3, and N is a positive integer; In response to the user's first operation on the first photo, the electronic device displays N first thumbnails; the N first thumbnails correspond one-to-one with the N frames of the first image; In response to a user's selection of a first target thumbnail from the N first thumbnails, the electronic device uses the first image corresponding to the first target thumbnail as a reference frame, merges M first images with the reference frame, and generates and saves a second photo; M = N-1, and M is a positive integer; wherein the M first images are all images in the N first images except the reference frame; wherein the second photo is a photo generated based on a second highlight frame, and the second highlight frame is the first image corresponding to the first target thumbnail selected by the user; when the first highlight frame is different from the second highlight frame selected by the user, the M first images include the first highlight frame.
2. The method according to claim 1, characterized in that, The first photo and the second photo are single photos; or, The first photo and the second photo are the covers of a live photo series with a duration of x seconds, where the number of frames i in the live photo series is greater than or equal to 3, i is a positive integer, and x is a positive number.
3. The method according to claim 1, characterized in that, The step of fusing the first image of M frames with the reference frame to generate the second photo includes: The electronic device performs weighted processing on the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the first image in the M-frame first image to generate the second photo.
4. The method according to claim 3, characterized in that, The electronic device performs weighted processing on the pixel values of each pixel in the reference frame and the pixel values of each pixel in each frame of the M-frame first image to generate the second photograph, including: The electronic device performs weighted processing on the pixel value of each pixel in the reference frame and the pixel value of each pixel in each frame of the first image in the M-frame first image to generate a first target image; The electronic device invokes a preset image algorithm to process the first target image and generate the second photo; wherein the image quality of the second photo is higher than that of the first photo.
5. The method according to any one of claims 1-4, characterized in that, The electronic device caches a second image at a first resolution; the second image corresponds to the second resolution; the method further includes: The electronic device receives a second operation from the user on the second photo; the second operation is used to set the resolution of the second photo to a third resolution; the third resolution is less than the first resolution and greater than the second resolution; In response to the second operation, the electronic device merges the second image at the first resolution and the second photograph at the second resolution to save the third photograph at the third resolution.
6. The method according to claim 5, characterized in that, The electronic device merges the second image at the first resolution and the second photograph at the second resolution, and saves the third photograph at the third resolution, including: The electronic device performs downsampling processing on the second image at the first resolution to generate the second image at the third resolution; The electronic device upsamples the second photo at the second resolution to generate the second photo at the third resolution. The electronic device merges the second image at the third resolution with the second photograph at the third resolution to generate the third photograph at the third resolution.
7. A photo processing method, characterized in that, Applied in electronic devices; the method includes: In response to a shooting command, the electronic device saves a first photo and caches N frames of the first image and a second image at a first resolution; wherein, the first photo is a photo generated based on a first highlight frame, and the first highlight frame is a frame identified by the electronic device from the N frames of the first image using an AI capture function; the N frames of the first image are captured by the electronic device within a preset time period before and after the user inputs the shooting command; N≥3, and N is a positive integer; In response to the user's first operation on the first photo, the electronic device displays N first thumbnails; the N first thumbnails correspond one-to-one with the N frames of the first image; In response to a user's selection of a first target thumbnail among the N first thumbnails, and a second operation on the first target thumbnail, the electronic device uses the first image corresponding to the first target thumbnail as a reference frame, and merges the M first images with the reference frame to generate a first target image; the first target image corresponds to a second resolution; wherein, the M first images are images among the N first images excluding the reference frame; The electronic device merges the first target image at the second resolution with the second image at the first resolution to save a second photo at the third resolution; wherein, the second operation is used to set the resolution of the second photo to the third resolution; the third resolution is less than the first resolution and greater than the second resolution; the second photo is a photo generated based on a second highlight frame, the second highlight frame being the first image corresponding to the first target thumbnail selected by the user; when the first highlight frame is different from the second highlight frame selected by the user, the M-frame first image includes the first highlight frame.
8. The method according to claim 7, characterized in that, The electronic device fuses the first target image at the second resolution with the second image at the first resolution, and saves the second photograph at the third resolution, including: The electronic device performs downsampling processing on the second image at the first resolution to generate the second image at the third resolution; The electronic device performs upsampling processing on the first target image at the second resolution to generate the first target image at the third resolution; The electronic device merges the first target image at the third resolution with the second image at the third resolution, and saves the second photograph at the third resolution.
9. An electronic device, characterized in that, include: Memory and one or more processors, camera; The memory stores computer program code, which includes computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1-6, or performs the method as described in claim 7 or 8.
10. A chip system, characterized in that, include: At least one processor and an interface; The interface is used to receive instructions and transmit them to the at least one processor; The at least one processor executes the instructions to cause the electronic device to perform the method as claimed in any one of claims 1-6, or to perform the method as claimed in claim 7 or 8.
11. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-6, or to perform the method as described in claim 7 or 8.
12. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-6, or to perform the method as described in claim 7 or 8.
Citation Information
Patent Citations
Image processing method, electronic device and nonvolatile computer readable storage medium
CN112367459A
Image processing method and device, electronic equipment and computer readable storage medium
CN115731143A
