Thumbnail generation method and apparatus

Abstract

This application provides a thumbnail generation method and apparatus, relating to the field of terminal technology. The method includes: in a multi-frame fusion scenario, since the first preview image is obtained by image fusion based on at least two images with different exposure times, the thumbnail obtained by processing the first preview image has good image quality.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a method and apparatus for generating thumbnails. Background Technology

[0002] With the popularization and development of electronic devices, people's functional needs for these devices are becoming increasingly diversified. For example, to meet users' needs for taking photos, many electronic devices support photo-taking functions and display a thumbnail of the photo in the lower left corner of the camera interface after a photo is taken. The content displayed in the thumbnail is consistent with the content in the photo, allowing users to confirm the photo's appearance through the thumbnail.

[0003] However, thumbnails often suffer from low image quality. Summary of the Invention

[0004] This application provides a thumbnail generation method and apparatus to improve the image quality of thumbnails.

[0005] In a first aspect, embodiments of this application provide a thumbnail generation method, the method comprising: responding to a first operation of taking a picture, an electronic device acquiring a first picture sequence and a first preview sequence, wherein any image in the first preview sequence is obtained by image fusion based on at least two images with different exposure times, and the first picture sequence includes a long frame sequence and a short frame sequence; the electronic device acquiring a first target image whose image quality meets preset requirements from the long frame sequence and the short frame sequence; the electronic device acquiring a first preview image from the first preview sequence, wherein the first preview image is an image in the first preview sequence that was captured at the same time as the first target image; and the electronic device processing the first preview image into a thumbnail.

[0006] In multi-frame fusion scenarios, since the first preview image is obtained by fusion of at least two images with different exposure times, the thumbnail obtained by processing the first preview image has good image quality.

[0007] In one possible implementation, the electronic device acquires a first target image whose image quality meets preset requirements from a long frame sequence and a short frame sequence, including: the electronic device acquires a first target image whose image quality meets preset requirements from a long frame sequence.

[0008] In multi-frame fusion scenarios, since the noise of long frame images is less than that of short frame images, long frame sequences with better image quality can be used for filtering to obtain captured images that meet preset requirements, and thumbnails with better image quality can be generated using the preview images corresponding to the captured images.

[0009] In one possible implementation, the image quality meeting the preset requirements includes one or more of the following: the time difference between the image capture time and the first operation time is less than a first threshold, the image focusing result is focusing completed or focusing locked, the image exposure result is exposure completed or exposure locked, or the image jitter is less than a second threshold.

[0010] Electronic devices can filter out a first target image that meets the image quality requirements by using multiple preset conditions, so as to achieve high-quality acquisition of the first target image. In this way, the first preview image acquired based on the first target image also has high image quality.

[0011] In one possible implementation, the first target image is any image in a first sequence, the first sequence being composed of images in the long frame sequence whose image quality meets preset requirements, and the number of images in the first sequence being less than or equal to the number of images in the long frame sequence; or, the first target image is an image that satisfies the following conditions: the time difference between the image capture time and the first operation is less than a first threshold, the image's focus result is focus complete or focus locked, the image's exposure result is exposure complete or exposure locked, and the image's jitter is less than a second threshold.

[0012] The electronic device can acquire any image from a first sequence that meets preset requirements as the first target image, or it can select the image with the highest image quality as the first target image.

[0013] In one possible implementation, the method further includes: when any image in the first preview sequence is in a first format, the electronic device converts the format of the first preview sequence from the first format to a second format, wherein the first format includes UBWC and the second format includes YUV.

[0014] This application does not limit the format of the first preview sequence. Even if the first preview image is in a first format or other formats, the electronic device can convert the format of the first preview image from the first format to the second format through the format conversion module.

[0015] In one possible implementation, the electronic device processes the first preview image into a thumbnail, including: the electronic device obtaining the thumbnail by sampling the first preview image; and the electronic device converting the format of the thumbnail from a second format to a third format, the third format including JPEG.

[0016] After implementing thumbnail processing, electronic devices can also convert the thumbnail format from the second format to the third format. The third format is not limited to JPEG, and this application embodiment does not limit it.

[0017] In one possible implementation, after the electronic device acquires the first image sequence and the first preview sequence, the state of any image in the first preview sequence is idle. After the electronic device acquires the first preview image from the first preview sequence, the method further includes: the electronic device sets the state of the first preview image to busy. After the electronic device processes the first preview image into a thumbnail, the method further includes: the electronic device releases the first preview image and sets the state of the first preview image to idle.

[0018] Electronic devices can manage images in a cache sequence by setting idle and busy states. Since image sequences in busy state cannot be released, the stability of the first preview image can be guaranteed during the process of processing it into a thumbnail.

[0019] In one possible implementation, the first photo sequence further includes a short frame sequence, and the method further includes: the electronic device performing image fusion on the long frame sequence and the short frame images to obtain the fused first photo sequence; the electronic device acquiring the first photo image from the first photo sequence and storing the first photo image in a photo album application.

[0020] Electronic devices can obtain a first image sequence by fusing long frame sequences and short frame sequences, and then select images that meet the requirements from the first image sequence through filtering by the electronic devices, and then store the multi-frame fused image.

[0021] In one possible implementation, the method further includes: in response to a first operation of taking a photo, the method further includes: in response to a user opening an HDR photo function from a camera application, the electronic device displays a photo-taking interface, the photo-taking interface including a photo-taking button; the first operation of taking a photo includes: in response to a first operation on the photo-taking button.

[0022] Electronic devices can enter a multi-frame fusion scene by enabling HDR photography, and acquire the first shooting sequence and the first preview sequence in the multi-frame fusion scene.

[0023] Secondly, embodiments of this application provide a thumbnail generation apparatus, which includes a display unit and a processing unit. The display unit is used to process the data display steps in the thumbnail generation apparatus, and the processing unit is used to process the data processing steps in the thumbnail generation apparatus.

[0024] In one possible implementation, the thumbnail generation apparatus may further include a storage unit, which may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data.

[0025] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory is used to store code instructions; and the processor is used to run the code instructions, causing the electronic device to perform the method described in the first aspect or any implementation thereof.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, cause a computer to perform the method described in the first aspect or any implementation thereof.

[0027] Fifthly, a computer program product comprising a computer program that, when run, causes a computer to perform the methods described in the first aspect or any implementation thereof.

[0028] It should be understood that the second to fifth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0029] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0030] Figure 2 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0031] Figure 3 A scenario diagram provided for an embodiment of this application;

[0032] Figure 4 A schematic diagram illustrating the principle of a thumbnail generation method provided in an embodiment of this application;

[0033] Figure 5 A schematic diagram illustrating the principle of another thumbnail generation method provided in this application embodiment;

[0034] Figure 6 A schematic diagram illustrating the module interaction of a thumbnail generation method provided in an embodiment of this application;

[0035] Figure 7 A schematic diagram illustrating the principle of a thumbnail generation method provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the structure of a thumbnail generation device provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0038] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms used in the embodiments of this application are explained. It should be understood that this explanation is for the purpose of more clearly explaining the embodiments of this application and does not necessarily constitute a limitation on the embodiments of this application.

[0039] 1. Previewing and capturing images

[0040] Preview images can be data captured in real-time by the camera of an electronic device and displayed in a preview screen. For example, when an electronic device receives a user's command to open a camera app, it can capture a preview image from the camera and display it in real-time in the camera app's preview screen.

[0041] A photographed image can be data acquired based on the shutter button on an electronic device. For example, when an electronic device receives a user's trigger action on the shutter button, it can acquire a photographed image taken at the moment the photo is taken using the camera.

[0042] 2. YUV

[0043] The YUV color space can be a color encoding method, where Y represents luminance and U and V represent chrominance. The YUV color space focuses on the visual sensitivity to luminance.

[0044] 3. RAW

[0045] RAW (Raw) images are raw images containing data processed from the image sensor of a digital camera, scanner, or film scanner. They are named as such because RAW images have not been processed, printed, or used for editing. RAW images contain the most original information about the image.

[0046] 4. High Dynamic Range (HDR)

[0047] HDR is a processing technique that enhances the brightness and contrast of images. Compared to ordinary images, HDR can provide more dynamic range and image detail. It uses images with the best detail corresponding to each exposure time to synthesize the final HDR image, which can better reflect the visual effects of the real environment.

[0048] One possible implementation for an electronic device to determine whether the current scene is an HDR scene is as follows: The electronic device downsamples the preview image by 4 times to obtain a preview thumbnail, and determines whether the proportion of the number of bright pixels in the preview thumbnail to the total number of pixels in the preview thumbnail is greater than a preset pixel threshold. The preview thumbnail can be obtained by storing every row of pixels in the frame corresponding to the preview image, keeping one row of pixels for every two rows. The bright pixels can be determined based on a grayscale threshold, which can be used to determine whether the current scene is a high dynamic range scene. The electronic device can determine whether the current scene is a high dynamic range scene based on one frame of data or multiple frames of data; this embodiment does not limit this.

[0049] 5. Electronic equipment

[0050] Electronic devices can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can include mobile phones with touchscreens, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments in this application do not limit the specific technologies or device forms used in the electronic devices.

[0051] 6. Other

[0052] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first value and the second value are only used to distinguish different values ​​and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that terms such as "first" and "second" do not necessarily imply that they are different.

[0053] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0055] To better understand the embodiments of this application, the structure of the electronic device according to the embodiments of this application is described below. For example, Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0056] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0057] The sensor module 180 may include a gyroscope sensor, which can be used to determine the motion posture of the electronic device 100.

[0058] In this embodiment, the electronic device 100 can determine its angular velocity around three axes (i.e., x, y, and z axes) using a gyroscope sensor. The gyroscope sensor can be used for image stabilization. For example, in response to a user's photo-taking operation, the gyroscope sensor can detect the angle of vibration of the electronic device 100, calculate the distance the lens module needs to compensate based on the angle, and then the lens can counteract the vibration of the electronic device 100 through reverse movement, thus achieving image stabilization. The electronic device can also calculate the amount of vibration based on the angle of vibration determined by the gyroscope sensor.

[0059] In possible implementations, sensor 180 may also include one or more of the following: pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, proximity sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, or bone conduction sensor, etc. Figure 1 (not shown in the text), and this application does not specifically limit this aspect in the embodiments.

[0060] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0061] Processor 110 may include one or more processing units. These processing units may be independent devices or integrated within one or more processors. Processor 110 is used to implement the processing steps in the thumbnail processing method described in the embodiments of this application.

[0062] The processor 110 may also include a memory for storing instructions and data; for example, the memory may be used to store preview images in a cache queue.

[0063] USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0064] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the charging management module 140 and the processor 110.

[0065] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0066] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), and other wireless communication technologies.

[0067] Electronic devices achieve display functions through GPUs, displays (194), and application processors.

[0068] Display screen 194 is used to display images, videos, etc. For example, an electronic device can display such as... Figure 1 The interface shown as a or b in the diagram.

[0069] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0070] Camera 193 is used to capture still images or videos. For example, in response to a user opening a camera app or taking a photo, the electronic device controls camera 193 to capture a raw image, and the electronic device obtains a preview image through pre-preview processing and post-preview processing of the raw image, and processes the preview image into a thumbnail.

[0071] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The internal memory 121 can be used to store computer executable program code, which includes instructions.

[0072] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0073] A touch sensor can be placed on the display screen 194, and the touch sensor and the display screen 194 together form a touch screen, also known as a "touchscreen".

[0074] Buttons 190 include a power button, volume buttons, etc. Indicator 192 may be an indicator light.

[0075] The software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0076] For example, Figure 2 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into multiple layers, from top to bottom: the application (APP) layer, the application framework (or FWK) layer, the hardware abstraction layer (HAL), and the kernel layer, etc. This application does not limit the specific layers.

[0077] The application layer may include a series of application packages. The application layer may include one or more of the following: camera, Bluetooth, or telephone applications, etc., which are not limited in this embodiment.

[0078] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes several predefined interfaces. It may include one or more of the following: camera FWK, surface flinger (SF), or buffer queue.

[0079] The camera FWK is used to enable image transfer between the application layer and the HAL, such as sending images generated by the camera application to the HAL, or returning thumbnails generated by the HAL to the camera application.

[0080] The main function of the display compositor is to receive graphics data buffers from multiple sources, composite them into layers, and then send them to the layer processing module in HAL.

[0081] The cache queue is used to cache and release preview images.

[0082] In possible implementations, the application framework layer may also include one or more of the following: a window manager, a content provider, a resource manager, a view system, or a notification manager, etc. Figure 2 (Not shown in the image).

[0083] The purpose of the hardware abstraction layer is to abstract hardware, providing a unified interface for upper-layer applications to query hardware devices, or to provide data storage services for upper-layer applications.

[0084] The hardware abstraction layer may include one or more of the following: a platform image processing module, a hardware composer (HWC), and a device image engine. The platform image processing module may be an image processing module provided by the chip, the HWC may be a layer processing module provided by the electronic device, and the device image engine may be an image processing module provided by the electronic device.

[0085] The platform's image processing module can be used to perform pre-processing and post-processing of images. For example, post-processing can include preview post-processing to generate preview images, and image capture post-processing to generate captured images. The processing steps in preview post-processing and image capture post-processing are different.

[0086] The device image engine may include: a preview image management module, a preview frame selection module, a processing engine, and a thumbnail processing module.

[0087] The preview image management module is used to cache preview images and release any preview image. Any preview image can correspond to a busy state or an idle state. Preview images in the busy state do not support release, while preview images in the idle state can support release.

[0088] The preview frame selection module is used to confirm the time of the photo capture from the preview image and to generate a first preview image for thumbnails.

[0089] The processing engine is used to process captured images and to select the desired first target image from multiple captured images.

[0090] The thumbnail processing module is used to perform thumbnail processing on images, such as obtaining thumbnails by shrinking a preview image of a frame and / or by image sampling.

[0091] In possible implementations, the hardware abstraction layer may also include a hardware editor (hardware composer, HWC), which mainly provides hardware support for the display compositor and supports functions such as layer compositing and display.

[0092] The kernel layer is the layer between hardware and software. It drives the hardware to function. The kernel layer can include one or more of the following: camera driver, display driver, or sensor driver, etc.

[0093] In possible implementations, the electronic device may include a hardware layer, which may include one or more of the following: a camera, a liquid crystal display (LCD), a graphics processing unit (GPU), or a central processing unit (CPU).

[0094] In this application embodiment, no specific limitations are made on the software layers involved in the software architecture, the modules contained in the layers, and the functions of the modules.

[0095] For example, Figure 3 This is a schematic diagram of a scenario provided for an embodiment of this application. Figure 3 In the corresponding embodiments, a mobile phone is used as an example for illustration. This example does not constitute a limitation on the embodiments of this application.

[0096] Electronic devices can display something like this when they detect that a user has opened the camera app. Figure 3 The interface shown in Figure 'a' may include: a camera button 301, a thumbnail 302, a preview screen 303, controls for switching cameras, and controls for indicating the use of the camera function. The camera application may also include other functions besides photography, such as supporting night scene shooting, portrait shooting, and video recording.

[0097] The preview screen 303 can display a preview image obtained by the electronic device in real time based on the camera. For example, the preview screen 303 may show a girl. The thumbnail 302 may be a thumbnail of the previous image obtained by the electronic device based on the camera application. For example, the thumbnail 302 may show a boy. The user can open the photo album application and display the photo image corresponding to the thumbnail 302 by triggering the operation of the thumbnail 302.

[0098] In response to the user's triggering of the camera button 301, the electronic device displays as follows: Figure 3 The interface shown in b can include a preview screen and a thumbnail 304, which can display a girl. Other content displayed on this interface is... Figure 3 The interface shown in 'a' is similar and will not be described again here.

[0099] like Figure 3 The interface shown in b can be a preview image obtained by the electronic device through pre-preview processing and post-preview processing of the original image, and a thumbnail image obtained by the electronic device through pre-photograph processing, post-photograph processing, and thumbnail processing of the original image. The preview image can be in YUV format, and the photographed image can be in RAW format.

[0100] For example, in combination Figure 4 The corresponding embodiments illustrate the thumbnail generation process. Figure 4 This is a schematic diagram illustrating the principle of a thumbnail generation method.

[0101] like Figure 4 As shown, in response to a user's triggering of the shutter button 301, the electronic device acquires the original image, which can be in RAW format. The electronic device can acquire a set of RAW format images at the moment of capture through pre- and post-processing of the original image, save the captured images, and then perform RAW-YUV conversion on the captured images to obtain YUV format images. Furthermore, through thumbnail processing and YUV-JPEG processing, it obtains thumbnails in JPEG format, which are then displayed. Figure 3 The bottom left corner of the interface shown by b.

[0102] However, the thumbnail generation time in the above methods is relatively long. For example, in the process of electronic devices processing photographed images into thumbnails, the RAW-YUV format conversion and YUV-JPEG format conversion of the photographed images is required, which wastes power and also makes the thumbnail generation time long.

[0103] Therefore, this application provides a thumbnail generation method that can generate thumbnails using preview images, reducing the need for image conversion from RAW-YUV format and decreasing thumbnail generation time.

[0104] For example, Figure 5 A schematic diagram illustrating the principle of another thumbnail generation method provided in this application embodiment. (See attached diagram.) Figure 5 As shown, in response to a user's triggering of the camera button 301, the electronic device acquires the original image. The electronic device can acquire multiple preview images through pre- and post-preview processing of the original image, and obtain a target preview image from the multiple preview images based on the best captured image. The electronic device then performs thumbnail processing and YUV-JPEG processing on the target preview image to obtain a thumbnail in JPEG format, which is then displayed. Figure 3 The bottom left corner of the interface shown by b.

[0105] Figure 5 The method shown reduces the conversion of images from RAW-YUV format and decreases thumbnail generation time. However, since the target preview image is a single frame, image quality cannot be guaranteed.

[0106] In view of this, embodiments of this application provide a thumbnail generation method. In a scenario of multi-frame image fusion, the electronic device can obtain a first preview sequence and a first image sequence. Since any image in the preview image is obtained by fusing at least two images with different exposure times, the thumbnail generated based on the preview image has better image quality.

[0107] Furthermore, in the process of selecting the first target image from the first image sequence, since the noise of any image in the long frame sequence of the first image sequence is relatively small, the electronic device can obtain the image with better image quality from the long frame sequence based on preset rules, and then the electronic device obtains the preview image corresponding to the image, and generates a thumbnail with better image quality through the preview image.

[0108] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other. The same or similar concepts or processes may not be described again in some embodiments.

[0109] For example, Figure 6 This is a schematic diagram illustrating the module interaction of a thumbnail generation method provided in an embodiment of this application. Figure 6 In a corresponding embodiment, the electronic device may include modules such as a camera application, a cache queue, a platform image processing module, a preview image management module, a preview frame selection module, a thumbnail processing module, a processing engine, and a camera driver.

[0110] like Figure 6 As shown, the thumbnail generation method may include the following steps:

[0111] S601, In response to the user's photo-taking operation, the camera driver will acquire the original image sequence and send it to the platform's image processing module.

[0112] The photo-taking operation can be targeted Figure 1 The trigger operation of the camera button shown in 'a' can also be a voice operation, gesture operation, etc. to achieve the camera taking, but this application embodiment does not limit this.

[0113] The original image sequence can be images captured by a camera, and any image in the original image sequence can be in RAW format.

[0114] Understandably, because electronic devices can display previews, generate captured images, and create thumbnails for HDR scenes, they can capture at least two images with different exposure levels, such as at least two images with different exposure times, in HDR scenarios. For example, in response to a user's triggering of the shutter button in an HDR shooting application, the camera driver obtains a sequence of raw images based on different shooting parameters and sends this sequence to the platform's image processing module. Alternatively, if the electronic device detects an HDR scene in the camera application, in response to the user's shooting action, the camera driver will acquire the sequence of raw images and send it to the platform's image processing module.

[0115] For example, let's take the acquisition of two images with different exposure times as an example. The number of original image sequences can be 2N (or 2M, where M is greater than N). Any group of images in the original image sequence can include at least two images with different exposure levels. For example, the at least two images with different exposure levels can be at least two frames of a long frame, a normal frame, or a short frame. The exposure time of the long frame can be greater than the exposure time of the short frame, and the exposure time of the normal frame can be greater than the exposure time of the short frame.

[0116] Combination Figure 7 The corresponding embodiment provides an example illustration of the original image sequence. Figure 7 This is a schematic diagram illustrating the principle of a thumbnail generation method provided in an embodiment of this application. Figure 7 As shown, the electronic device can obtain 18 sets of 36 original images. Taking one set of images as an example, for example, 83 and 83' can both be original images acquired by the electronic device at the same time. The exposure levels of 83 and 83' are different. For example, 83 can be a short frame and 83' can be a long frame.

[0117] Images in the original image sequence can be in the consumer queue, producer queue, and upcoming queue, respectively. For example, images 78 and 78'-86 and 86' can belong to the consumer queue, images 87 and 87'-91 and 91' can belong to the producer queue, and images 92 and 92'-95 and 95' can belong to the upcoming queue.

[0118] The images in the consumer queue can be images generated before the user takes a picture, or images acquired in advance; the images in the producer queue and the images in the queue to be generated can be images generated after the user takes a picture, or images acquired based on the steps shown in S601.

[0119] Understandably, the process of an electronic device acquiring a raw image sequence in an HDR scene can be as follows: Prior to S601, in response to a user opening a camera application, the camera driver could acquire a raw image sequence captured by the camera and perform scene recognition based on the raw image sequence. For example, if the electronic device determines that it is currently in an HDR scene, it can acquire at least two raw image sequences with different exposure levels based on the user's photo-taking operation.

[0120] S602, the platform image processing module performs image fusion processing and preview image processing on the original image sequence to obtain a second preview sequence, and performs image processing on the original image sequence to obtain a second image sequence.

[0121] The process of processing the original image sequence into a second preview sequence can be as follows: perform exposure fusion processing on each group of images in the original image sequence to obtain the fused image; perform pre-preview processing and post-preview processing on the fused image to obtain the second preview sequence after image processing.

[0122] Preview preprocessing can convert the fused image from RAW format to YUV format or other formats. For example, preview preprocessing may also include one or more of the following image preprocessing methods in addition to the above-mentioned RAW-YUV format conversion: bad pixel correction processing, RAW domain noise reduction processing, black level correction processing, optical shadow correction processing / automatic white balance processing, color interpolation processing, global tone mapping processing, color correction processing, or gamma correction processing, etc., which are not limited in this application embodiment.

[0123] Post-preview processing can perform image stabilization and / or skin beautification on the image processed before preview, but this application embodiment does not limit this.

[0124] The process of processing the original image sequence into a second photographic sequence can be as follows: perform pre-photograph processing and post-photograph processing on each image in the original image sequence to obtain the processed second photographic sequence.

[0125] Pre-processing for taking photos can include other image pre-processing methods besides RAW-YUV format conversion in preview pre-processing. Therefore, the image after pre-processing and any image in the second photo sequence can be in the RAW domain. The image processing methods in post-processing can be similar to or the same as those in preview post-processing, and will not be elaborated here.

[0126] The platform image processing module can process the original image sequence into a second preview sequence and a second photo sequence in S602, and then simultaneously execute the steps shown in S603 and S606.

[0127] It is understandable that there is a correspondence between the second preview sequence and the second photographed sequence obtained by processing the same original image sequence data. For example, electronic devices can indicate the above correspondence based on frame identifiers. Frame identifiers can be shooting time identifiers, frame numbers, or other identifiers.

[0128] S603, the platform image processing module caches the second preview sequence into the cache queue.

[0129] The cache queue is used to cache the second preview sequence. After receiving the second preview sequence, the cache queue can request memory to store the second preview sequence. The cache queue can set the state of any preview image in the second preview sequence to an idle state, and preview images in the idle state can be released.

[0130] The buffer queue can cache N frames of images. In a possible implementation, the buffer queue can also cache Q data points, where Q is greater than N. For example, in response to a user capturing a live image, the electronic device can cache Q image data points in the buffer queue. It is understood that the number of data points allowed to be cached in the buffer queue can vary depending on the scenario, and this embodiment does not impose any limitation on this.

[0131] S604, The preview image management module retrieves the first preview sequence from the cache queue.

[0132] The number of images in the first preview sequence can be less than the number of images in the second preview sequence.

[0133] After the preview image management module retrieves the first preview sequence from the cache queue, it does not store the preview images. This means the preview image management module follows a copy-free mechanism for preview images. In this scenario, the preview image management module does not need to allocate memory to store the retrieved preview images, saving memory and improving the overall performance of subsequent thumbnail generation based on the first preview image in the first preview sequence.

[0134] like Figure 7 As shown, the first preview sequence may include: 83a, 84a, 85a, 86a and 87a. Any preview image can be obtained by image fusion of two frames with different exposure levels in the original image sequence. For example, 83a can be obtained by image fusion of 83 and 83'.

[0135] S605, The preview image management module sends the first preview sequence to the preview frame selection module.

[0136] In possible implementations, any preview image in the first preview sequence can be in an image format other than YUV, such as Ultra Efficient Bit Width Compression (UBWC). In this scenario, the preview image management module can also include a format conversion module, which converts preview images in other formats (such as UBWC) into YUV format preview images. Therefore, this embodiment can adapt to more image processing platforms and does not limit the specific format of the preview images.

[0137] S606, the platform image processing module sends the first image sequence to the processing engine.

[0138] like Figure 7 As shown, the first image sequence may include: 85b and 85b', 86b and 86b', 90b and 90b', 91b and 91b', and 92b and 92b'. For example, 85b can be obtained by processing an image of 85, and 85b' can be obtained by processing an image of 85'.

[0139] It is understandable that the first image sequence can be selected by the electronic device from the second image sequence in response to the image capture operation.

[0140] S607, The processing engine acquires the first target image from the first image sequence.

[0141] Any image captured in the first shooting sequence can be in RAW format, and any image can carry one or more of the following shooting information: shooting time (or shutter speed), image shake, image auto exposure (AE) result (or simply exposure result), or image auto focus (AF) result (or simply focus result).

[0142] For example, the processing engine can acquire a first target image from a first image capture sequence based on preset requirements. For instance, the processing engine acquires one or more of the following: the difference between the shooting time and the shooting operation time of any image, the image focus result, the image exposure result, or the image jitter amount, and acquires an image from the first image capture sequence that meets one or more of the following preset requirements: for example, the difference between the shooting time and the shooting operation time is less than a first threshold, the image focus result is focus completed or focus locked, the image exposure result is exposure completed or exposure locked, or the image jitter amount is less than a second threshold.

[0143] Understandably, the processing engine can select a first sequence that meets preset requirements from the first image sequence, and the first target image can be any image in the first sequence. Alternatively, the processing engine can also select the image with the best image quality from the first image sequence as the first target image; for example, the first target image can be any image that meets all the conditions mentioned above in the preset requirements.

[0144] The amount of jitter can be determined based on an electronic image stabilization (EIS) algorithm.

[0145] like Figure 7 As shown, the electronic device can determine 85b as the first target image based on all images in the first imaging sequence.

[0146] It is understood that the first image capture sequence may include at least two image sequences with different exposure times, such as a long frame sequence and a short frame sequence. In this embodiment, the sequence in the first image capture is not limited. The electronic device can determine whether a preset requirement is met based on the shooting information of all images in the first image capture sequence, and then select the first target image from the first image capture sequence. Alternatively, the electronic device can also determine whether the preset requirement is met based on the shooting information of all images in the long frame sequence or the short frame sequence, and then select the first target image from the long frame sequence or the short frame sequence, thereby saving resource consumption during the selection process.

[0147] In one possible implementation, the electronic device could also obtain the first target image from a long frame sequence based on filtering within the first image capture sequence. It is understood that because the image noise in a long frame sequence is less than that in a short frame sequence, the images in the long frame sequence have higher quality, allowing the electronic device to utilize the long frame sequence to achieve high-quality and high-speed image filtering.

[0148] S608, the processing engine sends the first target image to the preview frame selection module.

[0149] The first target image can carry a frame identifier.

[0150] S609, The preview frame selection module uses the first target image to match the first preview image from the first preview sequence.

[0151] The preview frame selection module can obtain a first preview image corresponding to the first target image based on the frame identifier. Both the first preview image and the first target image can be obtained by processing the original image sequence acquired at the same time. For example... Figure 7 As shown, the first target image can be 85b' (or 85b), and based on the frame identifier of 85b' (or 85b), the first preview image 85a is determined to be obtained from the preview image.

[0152] The preview frame selection module can execute S610 and S611 synchronously after S609, or execute S610 and S611 sequentially. This application embodiment does not limit this.

[0153] S610, the preview frame selection module notifies the cache queue to set the status of the first preview image to busy.

[0154] The first preview image in a busy state is not allowed to be released. For example, the cache queue can store N preview images by default. When a new frame of data enters the cache queue, the cache queue can release the frame of data at the beginning of the queue. If the frame of data at the beginning of the queue is in a busy state, the cache queue can release the next frame of data in an idle state.

[0155] It is understandable that during the process of generating a thumbnail based on the first preview image, the first preview image is always in a busy state, meaning that the electronic device can keep the first preview image in a cache queue so that any module in the electronic device can retrieve it at any time.

[0156] S611, the preview frame selection module sends the first preview image to the thumbnail processing module.

[0157] S612, The thumbnail processing module processes the first preview image into a thumbnail.

[0158] The thumbnail processing method is as follows: the first preview image is reduced in size according to a preset sampling factor. In this embodiment of the application, the thumbnail processing method is not limited.

[0159] S613, the thumbnail processing module processes the first preview image in YUV format into JPEG format.

[0160] The thumbnail processing module can convert the format of the first preview image from YUV to JPEG or red, green, blue, RGB, etc., but this application embodiment does not limit this.

[0161] S614, The thumbnail processing module sends the thumbnail to the camera application.

[0162] For example, the thumbnail processing module can transmit the thumbnail from the HAL to the application layer via the camera FWK module in the application framework layer. Upon receiving the thumbnail, the camera application can generate a new layer containing the thumbnail, and then call modules such as the surface flinger in the application framework layer, the HWC in the HAL, and the display driver to display the thumbnail, enabling the electronic device to display as shown. Figure 1 The interface shown in b is shown in the image.

[0163] S615, the camera application notification cache queue releases the first preview image.

[0164] After S615, the cache queue can switch the state of the first preview image from busy to idle, so that the first preview image in the idle state can be released after a thumbnail is generated based on the first preview image.

[0165] Based on this, in the method for generating thumbnails based on preview images provided in the embodiments of this application, the electronic device can obtain a first preview image corresponding to the first target image from the first preview sequence when filtering a first target image that meets the preset requirements for image quality from the first shooting sequence. Since the first preview image can be obtained by image fusion based on at least two images with different exposure times, the first preview image has higher image quality, thereby improving the image quality of the thumbnail.

[0166] Understandable Figure 6 The sequential relationship between the steps is only an example and is not limited in this embodiment.

[0167] The above combination Figures 3-7 The methods provided in the embodiments of this application have been described. The apparatus for executing the above methods, provided in the embodiments of this application, is described below. Figure 8 As shown, Figure 8 This is a schematic diagram of a thumbnail generation device provided in an embodiment of this application. The thumbnail generation device may be an electronic device in the embodiment of this application, or it may be a chip or chip system within an electronic device.

[0168] like Figure 8 As shown, the thumbnail generation device 800 can be used in communication equipment, circuits, hardware components, or chips. The thumbnail generation device 800 includes a display unit 801 and a processing unit 802. The display unit 801 supports the display steps of the display method; the processing unit 802 supports the information processing steps of the thumbnail generation device 800.

[0169] In one possible implementation, the thumbnail generation device 800 may further include a communication unit 803, which supports the thumbnail generation device 800 in performing steps such as receiving or sending messages.

[0170] The thumbnail generation apparatus described in the embodiments of this application can all include Figure 8 The units described in the corresponding embodiments.

[0171] Specifically, the processing unit 802 can be integrated with the display unit 801, and the processing unit 802 and the display unit 801 may communicate with each other.

[0172] In one possible implementation, the thumbnail generation apparatus 800 may further include a storage unit 804. The storage unit 804 may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data.

[0173] The storage unit 804 can exist independently or be connected to the processing unit 802 via a communication bus. Alternatively, the storage unit 804 can be integrated with the processing unit 802.

[0174] Taking the thumbnail generation device 800 as an example, which may be a chip or chip system of the electronic device in the embodiments of this application, the storage unit 804 may store computer-executable instructions for the method of the electronic device, so that the processing unit 802 executes the method of the electronic device in the above embodiments. The storage unit 804 may be a register, cache, or random access memory (RAM), etc., and the storage unit 804 may be integrated with the processing unit 802. The storage unit 804 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 804 may be independent of the processing unit 802.

[0175] In one possible implementation, the thumbnail generation device 800 may further include a communication unit 803. The communication unit 803 supports interaction between the thumbnail generation device 800 and other devices. For example, when the thumbnail generation device 800 is an electronic device, the communication unit 803 may be a communication interface or interface circuit. When the thumbnail generation device 800 is a chip or chip system within an electronic device, the communication unit 803 may be a communication interface. For example, the communication interface may be an input / output interface, pins, or circuits.

[0176] The apparatus in this embodiment can be used to execute the steps performed in the above method embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0177] Figure 9 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application.

[0178] The electronic device includes a processor 901, a communication line 904, and at least one communication interface. Figure 9 (The example provided uses communication interface 903 as an example.)

[0179] The processor 901 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0180] Communication line 904 may include circuitry for transmitting information between the aforementioned components.

[0181] Communication interface 903 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0182] Possibly, the electronic device may also include a memory 902.

[0183] The memory 902 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 904. The memory may also be integrated with the processor.

[0184] The memory 902 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 901. The processor 901 executes the computer execution instructions stored in the memory 902 to implement the method provided in the embodiments of this application.

[0185] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0186] In a specific implementation, as one example, the processor 901 may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 in the CPU.

[0187] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 9 Processors 901 and 905 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0188] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0189] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0190] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0191] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0192] The above combinations should also be included within the scope of computer-readable media. The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0193] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A method for generating thumbnails, characterized in that, The method is applied to an electronic device, which includes: a cache queue, a platform image processing module, a preview image management module, a preview frame selection module, a thumbnail processing module, a processing engine, and a camera driver; the method includes: In response to the first operation of taking a picture, the camera driver sends the acquired raw image sequence to the platform image processing module; The platform's image processing module performs image fusion processing and preview image processing on the original image sequence to obtain a second preview sequence, and performs image capture processing on the original image sequence to obtain a second image capture sequence. The platform image processing module caches the second preview sequence into the cache queue; The preview image management module selects a first preview sequence from the second preview sequence in the cache queue based on a copy-free mechanism, and sends the first preview sequence to the preview frame selection module; the number of images in the first preview sequence is less than the number of images in the second preview sequence; any image in the first preview sequence is obtained by image fusion based on at least two images with different exposure times; The platform image processing module sends the first image sequence to the processing engine; the first image sequence includes a long frame sequence and a short frame sequence; the first image sequence is selected from the second image sequence in response to the first operation; The processing engine obtains a first target image whose image quality meets preset requirements from the long frame sequence and the short frame sequence; the first target image is any image in the first sequence, the first sequence is composed of images in the long frame sequence whose image quality meets preset requirements, and the number of images in the first sequence is less than or equal to the number of images in the long frame sequence. The processing engine sends the first target image to the preview frame selection module; The preview frame selection module obtains a first preview image from the first preview sequence and sends the first preview image to the thumbnail processing module; the first preview image is an image in the first preview sequence that was captured at the same time as the first target image; The thumbnail processing module processes the first preview image into a thumbnail.

2. The method according to claim 1, characterized in that, The image quality meets the preset requirements, including one or more of the following: the time difference between the image capture time and the first operation time is less than a first threshold, the image focus result is focus completed or focus locked, the image exposure result is exposure completed or exposure locked, or the image jitter is less than a second threshold.

3. The method according to claim 1, characterized in that, The method further includes: When any image in the first preview sequence is in the first format, the preview image management module converts the format of the first preview sequence from the first format to the second format. The first format includes UBWC, and the second format includes YUV.

4. The method according to claim 3, characterized in that, The thumbnail processing module processes the first preview image into a thumbnail, including: The thumbnail processing module obtains the thumbnail by sampling the first preview image; The thumbnail processing module converts the format of the thumbnail from the second format to a third format, the third format including JPEG.

5. The method according to any one of claims 1-4, characterized in that, After the cache queue receives the second preview sequence, the state of any image in the second preview sequence is idle. After the preview frame selection module obtains the first preview image from the first preview sequence, the method further includes: the cache queue sets the state of the first preview image to a busy state; After the thumbnail processing module processes the first preview image into a thumbnail, the method further includes: the cache queue releases the first preview image and sets the state of the first preview image to the idle state.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: the first operation in response to taking a picture, the method further includes: In response to a user activating the HDR shooting function from a camera application, the electronic device displays a shooting interface, which includes a shooting button; The first operation in response to taking a photo includes: the first operation in response to the photo button.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the computer to perform the method as described in any one of claims 1-6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it causes the computer to perform the method as described in any one of claims 1-6.

Images