Thumbnail generation method and device

Abstract

The embodiment of the invention provides a thumbnail generation method and device, and relates to the technical field of terminals, and the method comprises the steps: in a multi-frame fusion scene, a first preview image is obtained through image fusion based on at least two images with different exposure times, and the first preview image is obtained through image fusion; therefore, the thumbnail processed based on the first preview image has better image quality.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a method and apparatus for generating thumbnails. Background Art

[0002] With the popularization and development of electronic devices, people's functional requirements for electronic devices have become increasingly diverse. For example, to meet users' photography needs for electronic devices, many electronic devices support the photography function and support displaying a thumbnail corresponding to a photo in the lower left corner of the photography interface in response to a photography operation. The content displayed in the thumbnail is the same as the content in the photo, and users can determine the photography situation through the thumbnail.

[0003] However, usually, there is a problem that the image quality of the thumbnail is relatively low. Summary of the Invention

[0004] Embodiments of this application provide a method and apparatus for generating thumbnails, which are used to improve the image quality of thumbnails.

[0005] In a first aspect, an embodiment of this application provides a method for generating a thumbnail. The method includes: in response to a first operation of taking a photo, an electronic device obtains a first photo sequence and a first preview sequence, any image in the first preview sequence is obtained by fusing images with at least two different exposure times, and the first photo sequence includes a long-frame sequence and a short-frame sequence; the electronic device obtains a first target image with an image quality meeting a preset requirement from the long-frame sequence and the short-frame sequence; the electronic device obtains a first preview image from the first preview sequence, and the first preview image is an image in the first preview sequence with the same shooting time as the first target image; the electronic device processes the first preview image into a thumbnail.

[0006] In a multi-frame fusion scenario, since the first preview image is obtained by fusing images with at least two different exposure times, the thumbnail processed based on the first preview image has better image quality.

[0007] In a possible implementation manner, when the electronic device obtains a first target image with an image quality meeting a preset requirement from the long-frame sequence and the short-frame sequence, it includes: the electronic device obtains a first target image with an image quality meeting a preset requirement from the long-frame sequence.

[0008] In a multi-frame fusion scenario, since the noise of the long-frame image is less than the noise of the short-frame image, the long-frame sequence with better image quality can be used for screening to obtain a photo image with an image quality meeting the preset requirement, and a thumbnail with better image quality can be generated by using the preview image corresponding to the photo image.

[0009] In a possible implementation, the image quality meeting the preset requirements includes one or more of the following: the time difference between the shooting time of the image and the time of the first operation is less than a first threshold, the focusing result of the image is focusing completed or focus locked, the exposure result of the image is exposure completed or exposure locked, or the amount of jitter of the image is less than a second threshold.

[0010] The electronic device can screen out the first target image with image quality meeting the requirements through multiple conditions in the preset requirements, so as to achieve the acquisition of the first target image with high quality. In this way, the first preview image obtained based on the first target image also has high image quality.

[0011] In a possible implementation, the first target image is any image in a first sequence, and the first sequence is composed of images in the long frame sequence whose image quality meets the preset requirements. The number of images in the first sequence is less than or equal to the number of images in the long frame sequence; alternatively, the first target image is an image that meets the time difference between the shooting time of the image and the first operation being less than a first threshold, and the focusing result of the image is focusing completed or focus locked, and the exposure result of the image is exposure completed or exposure locked, and the amount of jitter of the image is less than a second threshold.

[0012] The electronic device can obtain any image from the first sequence that meets the preset requirements as the first target image, or can also select the image with the highest image quality as the first target image.

[0013] In a possible implementation, the method further includes: when the format of any image in the first preview sequence is a first format, the electronic device converts the format of the first preview sequence from the first format to a second format. The first format includes: UBWC, and the second format includes: YUV.

[0014] The embodiments of the present application do not limit the format of the first preview sequence. Even if the first preview image is in the first format or other formats, the electronic device can convert the format of the first preview image from the first format to the second format through the format conversion module.

[0015] In a possible implementation, the electronic device processes the first preview image into a thumbnail, including: the electronic device obtains the thumbnail through sampling processing of the first preview image; the electronic device converts the format of the thumbnail from the second format to a third format, and the third format includes: JPEG.

[0016] After the electronic device implements the thumbnail processing, it can also convert the format of the thumbnail from the second format to the third format. The third format is not limited to JPEG, and the embodiments of the present application do not limit this.

[0017] In a possible implementation, after the electronic device acquires the first photographing sequence and the first preview sequence, the state of any image in the first preview sequence is the idle state. After the electronic device acquires the first preview image from the first preview sequence, the method further includes: the electronic device sets the state of the first preview image to the busy state; after the electronic device processes the first preview image into a thumbnail, the method further includes: the electronic device releases the first preview image and sets the state of the first preview image to the idle state.

[0018] The electronic device can manage the images in the cache sequence by setting the idle state and the busy state. Since the image sequence in the busy state is not allowed to be released, the stability of the first preview image during the process of being processed into a thumbnail can be guaranteed.

[0019] In a possible implementation, the first photographing sequence further includes: a short frame sequence. The method further includes: the electronic device performs image fusion on the long frame sequence and the short frame images to obtain the first photographing sequence after fusion; the electronic device acquires the first photographing image from the first photographing sequence and stores the first photographing image in the album application.

[0020] The electronic device can obtain the first photographing sequence by fusing the long frame sequence and the short frame sequence, and can obtain the photographing images that meet the requirements from the first photographing sequence through the screening of the electronic device, and then store the multi-frame fused photographing images.

[0021] In a possible implementation, the method further includes: in response to the first photographing operation, the method further includes: in response to the operation of the user turning on the HDR photographing function from the camera application, the electronic device displays a photographing interface, and the photographing interface includes: a photographing button; the response to the first photographing operation includes: the response to the first operation on the photographing button.

[0022] The electronic device can enter the multi-frame fusion scenario by turning on the HDR photographing function, and can acquire the first photographing sequence and the first preview sequence in the multi-frame fusion scenario.

[0023] In a second aspect, an embodiment of the present application provides a thumbnail generation device. The thumbnail generation device includes a display unit and a processing unit. The display unit is used to process the steps of data display in the thumbnail generation device, and the processing unit is used to process the steps of data processing in the thumbnail generation device.

[0024] In a possible implementation, the thumbnail generation device may further include a storage unit. The storage unit may include one or more memories, and the memory may be a device or a component in one or more circuits for storing programs or data.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory is used to store code instructions; the processor is used to run the code instructions so that the electronic device executes the method described in the first aspect or any implementation manner of the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing instructions, which when executed, cause a computer to execute the method described in the first aspect or any implementation manner of the first aspect.

[0027] In a fifth aspect, a computer program product includes a computer program, which when run, causes a computer to execute the method described in the first aspect or any implementation manner of the first aspect.

[0028] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects obtained by each aspect and the corresponding feasible implementation manners are similar and will not be elaborated herein. Description of the Drawings

[0029] Figure 1 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application;

[0030] Figure 2 It is a schematic diagram of the software structure of an electronic device provided by an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of a scenario provided by an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the principle of a thumbnail generation method provided by an embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of the principle of another thumbnail generation method provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of the module interaction of a thumbnail generation method provided by an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of the principle of a thumbnail generation method provided by an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of the structure of a thumbnail generation device provided by an embodiment of the present application;

[0037] Figure 9 It is a schematic diagram of the hardware structure of another electronic device provided by an embodiment of the present application. Detailed Embodiments

[0038] To facilitate a clear description of the technical solutions of the embodiments of the present application, the terms involved in the embodiments of the present application are explained. It can be understood that this explanation is for a clearer explanation of the embodiments of the present application and does not necessarily constitute a limitation on the embodiments of the present application.

[0039] 1. Preview image and captured image

[0040] The preview image can be data captured in real time by the camera of the electronic device and used to be displayed in the preview screen. For example, when the electronic device receives an operation from the user to open the camera application, the electronic device can capture the preview image captured by the camera and display it in real time in the preview screen of the camera application.

[0041] The captured image can be data obtained based on the capture button in the electronic device. For example, when the electronic device receives a trigger operation from the user for the capture button, the electronic device can obtain the captured image obtained based on the camera at the moment of capture.

[0042] 2. YUV

[0043] The YUV color space can be a color encoding method. Y represents luminance, and U and V represent chrominance. The YUV color space focuses on the visual sensitivity to luminance.

[0044] 3. RAW

[0045] RAW domain image, that is, the raw image contains the data processed by the image sensor of a digital camera, scanner or film scanner. It is named like this because the RAW domain image has not been processed, printed or used for editing. The RAW domain image contains the most original information of the image.

[0046] 4. High dynamic range (HDR)

[0047] HDR is a processing technology to improve the brightness and contrast of images. Compared with ordinary images, HDR can provide more dynamic range and image details. By synthesizing the final HDR image using the images with the best details corresponding to each exposure time, it can better reflect the visual effect in the real environment.

[0048] A possible implementation for the electronic device to determine whether the current is an HDR scenario may be as follows: The electronic device downsamples the preview image by 4 times to obtain a preview thumbnail, and determines whether the ratio of the number of highlighted pixels in the preview thumbnail to the number of all pixels in the preview thumbnail is greater than a preset pixel threshold. Among them, the preview thumbnail can be obtained by retaining one row of pixel points out of every two rows of pixel points in the picture corresponding to the preview image, and storing every other column of the obtained pixel points in each row. The highlighted pixel can be determined based on the gray threshold of the pixel point, and the pixel threshold can be used to determine whether the current scenario is a high-dynamic range scenario. Among them, the electronic device can determine whether the current is a high-dynamic range scenario based on one frame of data or multiple frames of data, and this is not limited in the embodiments of the present application.

[0049] 5. Electronic device

[0050] The electronic device can also be referred to as a terminal, (terminal), user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device can be a mobile phone with a touch screen, smart TV, wearable device, tablet computer (Pad), computer with wireless transceiver function, virtual reality (VR) electronic device, augmented reality (AR) electronic device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic device.

[0051] 6. Others

[0052] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first value and the second value are only used to distinguish different values, and do not limit their order. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily limit being different.

[0053] It should be noted that in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0054] In this application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0055] In order to better understand the embodiments of this application, the structure of the electronic device in the embodiments of this application will be introduced below. Exemplarily, Figure 1 It is a schematic structural diagram of an electronic device provided by an embodiment of this application.

[0056] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0057] The sensor module 180 may include a gyroscope sensor, and the gyroscope sensor may be used to determine the motion posture of the electronic device 100.

[0058] In the embodiments of the present application, the electronic device can determine the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) through a gyroscope sensor. The gyroscope sensor can be used for anti-shake during shooting. For example, in response to a user's camera operation, the gyroscope sensor can detect the angle of jitter of the electronic device 100, calculate the distance that the lens module needs to compensate based on the angle, and then the lens cancels the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The electronic device can also calculate the amount of jitter based on the angle of jitter determined by the gyroscope sensor.

[0059] In possible implementation manners, the sensor 180 may further include one or more of the following: a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, or a bone conduction sensor, etc. ( Figure 1 not shown in the figure), and the embodiments of the present application do not make specific limitations thereon.

[0060] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0061] The processor 110 may include one or more processing units. Among them, different processing units may be independent devices or integrated in one or more processors. The processor 110 is used to implement the processing steps in the thumbnail processing method in the embodiments of the present application.

[0062] A memory may also be provided in the processor 110 for storing instructions and data. For example, the memory may be used to store preview images in the cache queue.

[0063] The USB interface 130 is an interface that complies with the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0064] The charging management module 140 is used to receive a charging input from a charger. The power management module 141 is used to connect the charging management module 140 to the processor 110.

[0065] The wireless communication function of the electronic device may be implemented through antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, a modulation and demodulation processor, and a baseband processor, etc.

[0066] The wireless communication module 160 may provide solutions for wireless communications applied to an electronic device, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), etc.

[0067] The electronic device implements a display function through a GPU, a display screen 194, an application processor, etc.

[0068] The display screen 194 is used to display images, videos, etc. For example, the electronic device may display an interface as shown by a or b in Figure 1 .

[0069] The electronic device may implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0070] The camera 193 is used to capture still images or videos. For example, in response to a user's operation of opening a camera application or in response to a user's shooting operation, the electronic device controls the camera 193 to collect a raw image. The electronic device obtains a preview image through pre-preview processing and post-preview processing of the raw image, and processes the preview image into a thumbnail.

[0071] The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the electronic device. The internal memory 121 may be used to store computer-executable program code, and the executable program code includes instructions.

[0072] The electronic device may implement an audio function through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor, etc. For example, music playback, recording, etc.

[0073] The touch sensor may be disposed on the display screen 194. The touch sensor and the display screen 194 form a touch screen, or a "touch control screen".

[0074] The keys 190 include a power-on key, a volume key, etc. The indicator 192 may be an indicator light.

[0075] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture, etc., which will not be elaborated here.

[0076] Exemplarily, Figure 2 is a schematic diagram of the software structure of an electronic device provided by an embodiment of the present application. As Figure 4 shown, the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers is through software interfaces. In some embodiments, the Android system is divided into multiple layers, from top to bottom are the application (APP) layer, the application framework (FWK) layer, the hardware abstraction layer (HAL), and the kernel layer, etc., which are not limited in the embodiments of the present application.

[0077] The application layer may include a series of application packages. The application layer may include one or more of the following: applications such as a camera, Bluetooth, or a phone, etc., which are not limited in the embodiments of the present application.

[0078] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined interfaces. The application framework layer may include one or more of the following: camera FWK, surface flinger (SF), or buffer queue.

[0079] The camera FWK is used to implement the transmission of images between the application layer and the HAL. For example, it sends the images generated by the camera application to the HAL, or returns the thumbnails generated by the HAL to the camera application.

[0080] The main function of the surface flinger is to receive graphic data buffers from multiple sources, perform layer composition on them, and then send them to the layer processing module in the HAL.

[0081] The buffer queue is used to implement the caching and release of preview images.

[0082] In possible implementation manners, the application framework layer may also include one or more of the following: window manager, content provider, resource manager, view system, or notification manager, etc. ( Figure 2 not shown in

[0083] The purpose of the hardware abstraction layer is to abstract the hardware, which can provide a unified interface for querying hardware devices for the upper-layer applications, or can also provide data storage services for the upper-layer applications.

[0084] The hardware abstraction layer may include one or more of the following: a platform image processing module, a hardware composer (HWC), and a device image engine. The platform image processing module may be an image processing module provided by a chip. The HWC may be a layer processing module provided by an electronic device. The device image engine may be an image processing module provided by an electronic device.

[0085] The platform image processing module can be used to implement pre-image processing and post-image processing for images. For example, post-image processing may include post-preview processing of an image to generate a preview image, and post-photographing processing of an image to generate a photographed image. The processing procedures in post-preview processing and post-photographing processing are different.

[0086] The device image engine may include: a preview image management module, a preview frame selection module, a processing engine, and a thumbnail processing module.

[0087] The preview image management module is used to implement caching processing of preview images and release any preview image. Any preview image can correspond to a busy state and an idle state. A preview image in the busy state does not support release, and a preview image in the idle state can support release.

[0088] The preview frame selection module is used to confirm the photographing moment from the preview images in the photographed image and generate a first preview image for generating a thumbnail.

[0089] The processing engine is used to implement processing of the photographed image and screen out a required first target image from multiple frames of photographed images.

[0090] The thumbnail processing module is used to implement thumbnail processing of images. For example, a thumbnail is obtained through processing such as shrinking a certain frame of preview image and / or image sampling.

[0091] In a possible implementation, the hardware abstraction layer may further include: a hardware composer (HWC). The hardware composer mainly provides hardware support for a display compositor and supports functions such as layer composition and display.

[0092] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make it work. The kernel layer may include one or more of the following: a camera driver, a display driver, or a sensor driver, etc.

[0093] In possible implementation manners, a hardware layer may be included in the electronic device. For example, the hardware layer may include one or more of the following: a camera, a liquid crystal display (LCD), a graphics processing unit (GPU), or a central processing unit (CPU), etc.

[0094] In the embodiments of the present application, the software layer involved in the software architecture, the modules included in the layer, and the functions of the modules are not specifically limited.

[0095] Exemplarily, Figure 3 FIG. is a schematic diagram of a scenario provided for the embodiments of the present application. In Figure 3 In the corresponding embodiment, taking the electronic device as a mobile phone as an example for illustration, this example does not limit the embodiments of the present application.

[0096] When the electronic device detects that the user opens the camera application, it may display an interface as shown in Figure 3 a in. The interface may include: a capture button 301, a thumbnail 302, a preview screen 303, a control for switching the camera, and a control for indicating the use of the capture function. The camera application may also have other functions in addition to the capture function, such as supporting multiple functions like night shooting, portrait shooting, video recording, etc.

[0097] In the preview screen 303, a preview image obtained by the electronic device in real time based on the camera may be displayed. For example, a girl is shown in the preview screen 303. The thumbnail 302 may be a thumbnail of the previous image obtained by the electronic device based on the camera application. For example, a boy may be shown in the thumbnail 302. The user can open the album application and display the captured image corresponding to the thumbnail 302 through a trigger operation on the thumbnail 302.

[0098] In response to the user's trigger operation on the capture button 301, the electronic device displays an interface as shown in Figure 3 b in. The interface may include: a preview screen and a thumbnail 304. A girl may be shown in the thumbnail 304. The other contents displayed in this interface are similar to those in the interface shown in Figure 3 a in and will not be described in detail here.

[0099] As shown in Figure 3 b in the interface, the preview screen may be obtained by the electronic device through pre-preview processing and post-preview processing of the original image. The thumbnail 304 may be obtained by the electronic device through pre-capture processing, post-capture processing, and thumbnail processing of the original image. The preview image may be in the YUV format, and the captured image may be in the RAW format.

[0100] Exemplarily, in combination with Figure 4 the corresponding embodiment, the generation process of the thumbnail is illustrated by way of example, Figure 4 which is a schematic diagram of the principle of a thumbnail generation method.

[0101] As Figure 4 shown in [reference], in response to a triggering operation of the user on the photographing button 301, the electronic device acquires an original image, and the format of the original image may be RAW. The electronic device can obtain a set of photographed images in RAW format at the moment of photographing through pre-processing before photographing and post-processing after photographing of the original image, save the photographed images, the electronic device performs RAW-YUV conversion processing on the photographed images to obtain photographed images in YUV format, and through thumbnail processing and YUV-JPEG processing, obtains a thumbnail in JPEG format, and then displays it at Figure 3 the lower left corner of the interface shown in b of [reference].

[0102] However, in the above method, the thumbnail generation time is relatively long. For example, in the method of processing the photographed image into a thumbnail by the electronic device, it is necessary to perform RAW-YUV format conversion and YUV-JPEG format conversion processing on the photographed image, which wastes power consumption and also makes the thumbnail generation time relatively long.

[0103] Therefore, the embodiment of the present application provides a thumbnail generation method, which can realize generating a thumbnail by using a preview image, reduce the conversion of the image from RAW-YUV format, and reduce the thumbnail generation duration.

[0104] Exemplarily, Figure 5 a schematic diagram of the principle of another thumbnail generation method provided by the embodiment of the present application. As Figure 5 shown in [reference], in response to a triggering operation of the user on the photographing button 301, the electronic device acquires an original image. The electronic device can obtain multiple frames of preview images through pre-processing before preview and post-processing after preview of the original image, and obtain a target preview image from the multiple frames of preview images based on the best photographed image. The electronic device performs thumbnail processing and YUV-JPEG processing on the target preview image to obtain a thumbnail in JPEG format, and then displays it at Figure 3 the lower left corner of the interface shown in b of [reference].

[0105] Figure 5 The method shown in [reference] reduces the conversion of the image from RAW-YUV format and reduces the thumbnail generation duration. However, since the target preview image is a single-frame image, the image quality cannot be guaranteed.

[0106] In view of this, an embodiment of the present application provides a thumbnail generation method. In a scenario of multi-frame photo fusion, an electronic device can obtain a first preview sequence and a first photo sequence. Since any image in the preview images is obtained by fusing at least two images with different exposure times, the thumbnail generated based on the preview images has better image quality.

[0107] Furthermore, in the process of screening out the first target image from the first photo sequence, since any image in the long-frame sequence of the first photo sequence has less noise, the electronic device can obtain a photo image with better image quality from the long-frame sequence based on a preset rule. Then, the electronic device obtains the corresponding preview image of the photo image, and generates a thumbnail with better image quality through the preview image.

[0108] The following uses specific embodiments to detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. These several specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, they may not be repeated in some embodiments.

[0109] Exemplarily, Figure 6 is a schematic diagram of module interaction for a thumbnail generation method provided by an embodiment of the present application. In Figure 6 the corresponding embodiment, the electronic device may include: a camera application, a cache queue, a platform image processing module, a preview image management module, a preview frame selection module, a thumbnail processing module, a processing engine, and a camera driver, etc.

[0110] As Figure 6 shown, the thumbnail generation method may include the following steps:

[0111] S601. In response to a user's photo-taking operation, the camera driver sends the acquired original image sequence to the platform image processing module.

[0112] The photo-taking operation may be a triggering operation for the photo-taking button shown in a of Figure 1 , or it may also be a voice operation, a gesture operation, etc. for realizing photo-taking. In the embodiments of the present application, this is not limited.

[0113] The original image sequence may be images collected by the camera, and the format of any image in the original image sequence may be RAW format.

[0114] It can be understood that since the electronic device can implement the preview screen display, photo image generation, and thumbnail generation in the HDR scenario, in the case of the HDR scenario, the electronic device can capture at least two images with different exposure levels, such as at least two images with different exposure times. For example, in response to the user's trigger operation on the capture button in the HDR photo mode of the camera application, the camera driver obtains a raw image sequence based on different shooting parameters and sends the obtained raw image sequence to the platform image processing module. Alternatively, when the electronic device detects an HDR scenario in the camera application and in response to the user's capture operation, the camera driver sends the obtained raw image sequence to the platform image processing module.

[0115] For example, taking the example of obtaining two images with different exposure times for illustration. The number of the raw image sequences can be 2N (or 2M, where M is greater than N). Any group of images in the raw image sequences can include: at least two images with different exposure levels. For example, at least two images with different exposure levels can be at least two of a long frame, a normal frame, or a short frame. The exposure time of the long frame can be greater than that of the short frame, and the exposure time of the normal frame can be greater than that of the short frame.

[0116] Combined with Figure 7 the corresponding embodiments, the raw image sequences are illustrated by examples. Figure 7 is a schematic diagram of the principle of a thumbnail generation method provided by an embodiment of the present application. As Figure 7 shown, the electronic device can obtain 18 groups of a total of 36 raw images. Taking one group of images for illustration, for example, both 83 and 83' can be raw images simultaneously obtained by the electronic device. The exposure levels of 83 and 83' are different. For example, 83 can be a short frame and 83' can be a long frame.

[0117] The images in the raw image sequences can be respectively in: a consumer queue, a producer queue, and an upcoming queue. For example, the images among 78 and 78'-86 and 86' can belong to the consumer queue, the images among 87 and 87'-91 and 91' can belong to the producer queue, and the images among 92 and 92'-95 and 95' can belong to the upcoming queue.

[0118] Among them, the images in the consumer queue can be the images generated before the user's capture operation, or can also be understood as the images obtained in advance; the images in the producer queue and the images in the upcoming queue can be the images generated after the user's capture operation, or can also be understood as the images obtained based on the steps shown in S601.

[0119] It can be understood that the process of an electronic device acquiring an original image sequence in an HDR scenario can be as follows. Before S601, in response to a user's operation of opening the camera application, the camera driver can obtain the original image sequence collected based on the camera, and perform scene recognition based on the original image sequence. For example, when the electronic device determines that it is currently in an HDR scenario, based on the user's photographing operation, it can obtain at least two original image sequences with different exposure levels.

[0120] S602. The platform image processing module performs image fusion processing and preview image processing on the original image sequence to obtain a second preview sequence, and performs photographing image processing on the original image sequence to obtain a second photographing sequence.

[0121] The process of processing the original image sequence into a second preview sequence can be: performing exposure fusion processing on each group of images in the original image sequence to obtain a fused image; respectively performing pre-preview processing and post-preview processing on the fused image to obtain the second preview sequence after image processing.

[0122] Pre-preview processing can be used to convert the fused image from the RAW format to formats such as the YUV format. For example, pre-preview processing may further include one or more of the following image pre-processing methods in addition to the above RAW-YUV format conversion: dead pixel correction processing, RAW domain noise reduction processing, black level correction processing, optical shadow correction processing / auto white balance processing, color interpolation processing, global tone mapping processing, color correction processing, or gamma correction processing, etc., which are not limited in the embodiments of the present application.

[0123] Post-preview processing can be used to perform image anti-shake processing and / or beauty skin care processing on the image after pre-preview processing, etc., which are not limited in the embodiments of the present application.

[0124] The process of processing the original image sequence into a second photographing sequence can be: respectively performing pre-photographing processing and post-photographing processing on each image in the original image sequence to obtain the second photographing sequence after image processing.

[0125] Pre-photographing processing may include other image pre-processing methods in addition to the RAW-YUV format conversion in pre-preview processing, so the image after pre-photographing processing and any image in the second photographing sequence can be in the RAW domain. The image processing method in post-photographing processing can be similar to or the same as the method in post-preview processing, which will not be elaborated here.

[0126] After processing the original image sequence into a second preview sequence and a second photographing sequence in S602, the platform image processing module can simultaneously execute the steps shown in S603 and S606.

[0127] It can be understood that there is a corresponding relationship between the second preview sequence and the second photo-taking sequence obtained based on the same original image sequence data. For example, the electronic device can indicate the above corresponding relationship based on the frame identifier. The frame identifier can be a shooting time identifier, a frame number, or other identifiers, etc.

[0128] S603. The platform image processing module caches the second preview sequence into the cache queue.

[0129] The cache queue is used to cache the second preview sequence. After receiving the second preview sequence, the cache queue can apply for memory to store the second preview sequence. The cache queue can set the status of any preview image in the second preview sequence to the idle state. The preview image in the idle state is allowed to be released.

[0130] The cache queue can implement caching of N frames of images. In a possible implementation, the cache queue can also implement caching of Q data, where Q is greater than N. For example, in response to the user's operation of shooting a live image, the electronic device can cache Q image data in the cache queue. It can be understood that the number of data allowed to be cached in the cache queue can vary according to different scenarios, and this is not limited in the embodiments of the present application.

[0131] S604. The preview image management module obtains the first preview sequence from the cache queue.

[0132] The number of images in the first preview sequence can be less than the number of images in the second preview sequence.

[0133] After the preview image management module obtains the first preview sequence from the cache queue, it does not store the preview images, that is, the preview image management module follows a copy-free mechanism for preview images. In this scenario, the preview image management module does not need to apply for memory to store the obtained preview images, which can save memory and improve the overall performance of generating thumbnails based on the first preview images in the first preview sequence.

[0134] As Figure 7 shown, the first preview sequence can include: 83a, 84a, 85a, 86a, and 87a. Any preview image can be obtained by fusing two images with different exposure degrees in the original image sequence. For example, 83a can be obtained by fusing 83 and 83'.

[0135] S605. The preview image management module sends the first preview sequence to the preview frame selection module.

[0136] In a possible implementation, any preview image in the first preview sequence can also be in an image format other than the YUV format. For example, the preview image can be in an ultra-efficient bit width compression (UBWC) format or other formats. In this scenario, the preview image management module can further include a format conversion module, which is used to convert the preview image in other formats (such as the UBWC format) into a preview image in the YUV format. Therefore, the embodiments of the present application can be adapted to more image processing platforms and do not limit the specific format of the preview image.

[0137] S606. The platform image processing module sends the first photographing sequence to the processing engine.

[0138] As Figure 7 shown, the first photographing sequence may include: 85b and 85b', 86b and 86b', 90b and 90b', 91b and 91b', 92b and 92b'. For example, 85b may be obtained by performing photographing image processing on 85, and 85b' may be obtained by performing photographing image processing on 85'.

[0139] It can be understood that the first photographing sequence may be screened by the electronic device from the second photographing sequence in response to a photographing operation.

[0140] S607. The processing engine obtains a first target image from the first photographing sequence.

[0141] Any photographing image in the first photographing sequence can be in the RAW format, and any photographing image can carry one or more shooting information items such as the shooting time (or shutter time), the image shake amount, the image auto exposure (AE) result (or simply the exposure result), or the image auto focus (AF) result (or simply the focus result).

[0142] Exemplarily, the processing engine can obtain the first target image from the first photographing sequence based on a preset requirement. For example: The processing engine obtains one or more of the difference between the shooting time of any photographing image and the time of the photographing operation, the image focus result, the image exposure result, or the image shake amount, and obtains an image that meets one or more of the following preset requirements from the first photographing sequence. For example, the difference between the shooting time and the time of the photographing operation is less than a first threshold, the focus result of the image is focus completed or focus locked, the exposure result of the image is exposure completed or exposure locked, or the image shake amount is less than a second threshold.

[0143] It can be understood that the processing engine can obtain a first sequence that meets the preset requirements from the first photographing sequence, and the first target image can be any image in the first sequence. Alternatively, the processing engine can also obtain the image with the best image quality from the first photographing sequence as the first target image. For example, the first target image can be an image that meets all the conditions in the above preset requirements.

[0144] Among them, the amount of jitter can be determined based on the electronic image stabilization (EIS) algorithm.

[0145] As Figure 7 shown, the electronic device can determine that 85b is the first target image based on all the images in the first photographing sequence.

[0146] It can be understood that the first photographing sequence can include at least two image sequences with different exposure times. For example, it includes a long-frame sequence and a short-frame sequence. In the embodiments of the present application, the sequences in the first photographed images are not limited. The electronic device can determine whether the preset requirements are met based on the shooting information of all the images in the first photographing sequence, and then screen out the first target image from the first photographing sequence. Alternatively, the electronic device can also determine whether the preset requirements are met based on the shooting information of all the images in the long-frame sequence or the short-frame sequence, and then screen out the first target image from the long-frame sequence or the short-frame sequence to save the resource occupancy during the screening process.

[0147] In a possible implementation, the electronic device can also obtain the first target image from the long-frame sequence based on the screening of the long-frame sequence in the first photographing sequence. It can be understood that since the image noise in the long-frame sequence is less than the image noise in the short-frame sequence, the quality of the images in the long-frame sequence is relatively high, enabling the electronic device to utilize the long-frame sequence to achieve high-quality and high-speed image screening.

[0148] S608. The processing engine sends the first target image to the preview frame selection module.

[0149] The first target image can carry a frame identifier.

[0150] S609. The preview frame selection module uses the first target image to match the first preview image from the first preview sequence.

[0151] The preview frame selection module can obtain the first preview image corresponding to the first target image based on the frame identifier. The first preview image and the first target image can be processed from the original image sequence obtained at the same moment. As Figure 7 shown, the first target image can be 85b’ (or 85b). Based on the frame identifier of 85b’ (or 85b), it is determined to obtain the first preview image 85a from the preview images.

[0152] The preview frame selection module may synchronously execute S610 and S611 after S609, or execute S610 and S611 successively. This application embodiment does not make any limitations in this regard.

[0153] S610. The preview frame selection module notifies the cache queue to set the status of the first preview image to busy.

[0154] The first preview image in the busy state is not allowed to be released. Exemplarily, the cache queue may default to storing N preview images. When a new frame of data enters the cache queue, the cache queue may release a frame of data at the starting position of the queue. If the status of a frame of data at the starting position of the queue is busy, the cache queue may release the next frame of data in the idle state at the starting position.

[0155] It can be understood that during the process of generating a thumbnail based on the first preview image, the status of the first preview image has been in the busy state, that is, the electronic device can keep the first preview image cached in the cache queue so that any module in the electronic device can obtain it at any time.

[0156] S611. The preview frame selection module sends the first preview image to the thumbnail processing module.

[0157] S612. The thumbnail processing module processes the first preview image into a thumbnail.

[0158] The thumbnail processing method is: downsizing and sampling the first preview image according to a preset sampling multiple. This application embodiment does not make any limitations on the thumbnail processing method.

[0159] S613. The thumbnail processing module processes the first preview image in YUV format into JPEG format.

[0160] The thumbnail processing module can convert the format of the first preview image from YUV to JPEG or red, green, blue (RGB), etc. This application embodiment does not make any limitations in this regard.

[0161] S614. The thumbnail processing module sends the thumbnail to the camera application.

[0162] Exemplarily, the thumbnail processing module may transmit the thumbnail from the HAL to the application layer via the camera FWK module in the application framework layer, etc. After receiving the thumbnail, the camera application may generate a new layer containing the thumbnail, and then call modules such as surface flinger in the application framework layer, HWC in the HAL, and the display driver, etc., to implement the display of the thumbnail, so that the electronic device displays asFigure 1 The interface shown as b in

[0163] S615, the camera application notification cache queue releases the first preview image.

[0164] After S615, the cache queue can switch the status of the first preview image from the busy state to the idle state. In this way, after generating a thumbnail based on the first preview image, the first preview image in the idle state can be allowed to be released.

[0165] Based on this, in a method for generating a thumbnail based on a preview image provided by an embodiment of the present application, when an electronic device filters a first target image that meets the preset requirements for image quality from a first photographing sequence, the electronic device can obtain a first preview image corresponding to the first target image from a first preview sequence. Since the first preview image can be obtained by fusing images with at least two different exposure times, the first preview image has relatively high image quality, thereby improving the image quality of the thumbnail.

[0166] It can be understood that Figure 6 the order relationship between the steps in

[0167] is only taken as an example, and the embodiments of the present application do not limit this. Figures 3 - 7 The method provided by the embodiments of the present application has been described above in conjunction with Figure 8 . Next, a device for executing the above method provided by the embodiments of the present application will be described. As Figure 8 shown,

[0168] is a schematic structural diagram of a thumbnail generation device provided by an embodiment of the present application. The thumbnail generation device may be the electronic device in the embodiment of the present application, or a chip or a chip system within the electronic device. Figure 8 As

[0169] shown, the thumbnail generation device 800 may be used in a communication device, a circuit, a hardware component, or a chip. The thumbnail generation device 800 includes: a display unit 801 and a processing unit 802. Among them, the display unit 801 is used to support the display steps executed by the display method; the processing unit 802 is used to support the thumbnail generation device 800 to execute the information processing steps.

[0170] In a possible implementation manner, the thumbnail generation device 800 may further include a communication unit 803, and the communication unit 803 is used to support the thumbnail generation device 800 to execute steps such as receiving or sending messages. Figure 8 The units described in the corresponding embodiments.

[0171] Specifically, the processing unit 802 and the display unit 801 may be integrated together, and communication may occur between the processing unit 802 and the display unit 801.

[0172] In a possible implementation, the thumbnail generation device 800 may further include: a storage unit 804. Among them, the storage unit 804 may include one or more memories, and the memory may be a device or a component in one or more devices or circuits for storing programs or data.

[0173] The storage unit 804 may exist independently and be connected to the processing unit 802 through a communication bus. The storage unit 804 may also be integrated with the processing unit 802.

[0174] Taking the thumbnail generation device 800 as an example of a chip or a chip system of the electronic device in the embodiment of the present application, the storage unit 804 may store computer-executable instructions of the method of the electronic device, so that the processing unit 802 executes the method of the electronic device in the above embodiment. The storage unit 804 may be a register, a cache, or a random access memory (RAM), etc., and the storage unit 804 may be integrated with the processing unit 802. The storage unit 804 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, and the storage unit 804 may be independent of the processing unit 802.

[0175] In a possible implementation, the thumbnail generation device 800 may further include: a communication unit 803. Among them, the communication unit 803 is used to support the interaction between the thumbnail generation device 800 and other devices. Exemplarily, when the thumbnail generation device 800 is an electronic device, the communication unit 803 may be a communication interface or an interface circuit. When the thumbnail generation device 800 is a chip or a chip system inside an electronic device, the communication unit 803 may be a communication interface. For example, the communication interface may be an input / output interface, a pin, or a circuit, etc.

[0176] The device in this embodiment can correspondingly be used to execute the steps performed in the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0177] Figure 9 It is a schematic diagram of the hardware structure of another electronic device provided by the embodiment of the present application.

[0178] The electronic device includes a processor 901, a communication line 904, and at least one communication interface ( Figure 9 illustrated by taking the communication interface 903 as an example).

[0179] The processor 901 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the solution of the present application.

[0180] The communication line 904 may include a circuit for transmitting information between the above components.

[0181] The communication interface 903 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0182] Possibly, the electronic device may further include a memory 902.

[0183] The memory 902 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor through the communication line 904. The memory may also be integrated with the processor.

[0184] Among them, the memory 902 is used to store computer execution instructions for executing the solution of the present application and is controlled by the processor 901 for execution. The processor 901 is used to execute the computer execution instructions stored in the memory 902, thereby implementing the method provided by the embodiments of the present application.

[0185] Possibly, the computer execution instructions in the embodiments of the present application may also be referred to as application code, and the embodiments of the present application do not make specific limitations thereto.

[0186] In a specific implementation, as an embodiment, the processor 901 may include one or more CPUs, such as Figure 9 CPU0 and CPU1 in

[0187] In a specific implementation, as an embodiment, the electronic device may include multiple processors, such as Figure 9 processor 901 and processor 905 in

[0188] In the above embodiment, the instructions stored in the memory for the processor to execute may be implemented in the form of a computer program product. Among them, the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0189] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center including one or more integrated available media. For example, the available medium may include magnetic media (such as floppy disks, hard disks, or magnetic tapes), optical media (such as digital versatile discs (DVDs)), or semiconductor media (such as solid state disks (SSDs)).

[0190] The embodiments of the present application also provide a computer-readable storage medium. The methods described in the above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium accessible by a computer.

[0191] As a possible design, a computer-readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; a computer-readable medium may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting wire may also be appropriately referred to as a computer-readable medium. For example, if software is transmitted using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) from a website, server, or other remote source, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used herein, magnetic disks and optical discs include optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where magnetic disks typically reproduce data magnetically, while optical discs use lasers to optically reproduce data.

[0192] The above combinations should also be included within the scope of computer-readable media. The above is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, and all should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

[0193] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

Claims

1. A method for generating a thumbnail, characterized in that, the method includes: In response to a first operation of taking a photo, an electronic device acquires a first photo sequence and a first preview sequence. Any image in the first preview sequence is obtained by fusing images with at least two different exposure times, and the first photo sequence includes a long-frame sequence and a short-frame sequence; The electronic device acquires a first target image whose image quality meets a preset requirement from the long-frame sequence and the short-frame sequence; The electronic device acquires a first preview image from the first preview sequence, and the first preview image is an image in the first preview sequence that has the same shooting time as the first target image; The electronic device processes the first preview image into a thumbnail.

2. The method according to claim 1, characterized in that, The electronic device acquires a first target image whose image quality meets a preset requirement from the long-frame sequence and the short-frame sequence, including: The electronic device acquires the first target image whose image quality meets the preset requirement from the long-frame sequence.

3. The method according to claim 1 or 2, characterized in that, That the image quality meets the preset requirement includes one or more of the following: The time difference between the shooting time of the image and the time of the first operation is less than a first threshold, the focusing result of the image is focusing completed or focus locked, the exposure result of the image is exposure completed or exposure locked, or the amount of jitter of the image is less than a second threshold.

4. The method according to claim 3, characterized in that, The first target image is any image in a first sequence, and the first sequence is composed of images in the long-frame sequence whose image quality meets the preset requirement, and the number of images in the first sequence is less than or equal to the number of images in the long-frame sequence; Or, the first target image is an image that satisfies that the time difference between the shooting time of the image and the first operation is less than the first threshold, and the focusing result of the image is focusing completed or focus locked, and the exposure result of the image is exposure completed or exposure locked, and the amount of jitter of the image is less than the second threshold.

5. The method according to any one of claims 1-4, characterized in that, the method further includes: When the format of any image in the first preview sequence is a first format, the electronic device converts the format of the first preview sequence from the first format to a second format, the first format includes: UBWC, and the second format includes: YUV.

6. The method according to claim 5, characterized in that, The electronic device processes the first preview image into a thumbnail, including: The electronic device obtains the thumbnail through sampling processing of the first preview image; The electronic device converts the format of the thumbnail from the second format to a third format, and the third format includes: JPEG.

7. The method according to any one of claims 1-6, characterized in that, After the electronic device obtains the first photographing sequence and the first preview sequence, the state of any image in the first preview sequence is the idle state. After the electronic device obtains a first preview image from the first preview sequence, the method further includes: the electronic device sets the state of the first preview image to the busy state. After the electronic device processes the first preview image into a thumbnail, the method further includes: the electronic device releases the first preview image and sets the state of the first preview image to the idle state.

8. The method according to any one of claims 1-7, wherein, the method further includes: in response to the first operation for taking a photo, the method further includes: In response to an operation by the user to turn on the HDR photo-taking function in the camera application, the electronic device displays a photo-taking interface, and the photo-taking interface includes: a photo-taking button; The first operation in response to taking a photo includes: responding to the first operation on the photo-taking button.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, when the processor executes the computer program, the electronic device is caused to execute the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, wherein, when the computer program is executed by a processor, the computer is caused to execute the method according to any one of claims 1-8.

Images