Method for capturing image during video recording, and electronic device and readable medium

Abstract

Provided in the present application are a method for capturing an image during video recording, and an electronic device and a readable medium. The method comprises: displaying a video-recording interface of a camera application, wherein the dynamic range of an image displayed in the video-recording interface is greater than a first value, the first value indicates that the image displayed in the video-recording interface is a non-low-dynamic-range image, and the image displayed in the video-recording interface is stored in a cache region; and in response to a capture instruction during video recording, using as a captured image the image that is in the cache region and corresponds to a capture moment. A cache region stores an image displayed in a video-recording interface, that is, an image collected by a camera is stored in the cache region. The image is greater than a first value, and the first value indicates that the image is a non-low-dynamic-range image, indicating that the camera outputs high-dynamic-range (HDR) images and the video-recording interface is an HDR video-recording interface. When a capture instruction is formed during video recording, the image that is in the cache region and corresponds to a capture moment is used as a captured image, which capture moment is a moment corresponding to the capture instruction, thereby ensuring that ZSL requirements for image capture are met.

Description

Method for capturing images in video recording, electronic device and readable medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on December 26, 2023, with application number 202311816888.1 and invention name “Method for capturing images in video recording, electronic device and readable medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of photographing technology, and in particular to a method for capturing an image in a video, an electronic device, a computer program product, and a computer-readable storage medium. Background Art

[0003] High Dynamic Range (HDR) images can display the bright and dark parts of the picture with higher quality, provide image details, and better reflect the visual effects in the real environment.

[0004] Currently, electronic devices can switch between HDR and non-HDR modes during single-camera recording. The raw image output by the camera during recording can also be used as a snapshot image. When the camera is controlled to output a raw image in HDR mode, the output raw image becomes an HDR raw image, allowing the camera to capture images with HDR effects during recording. However, the images captured by electronic devices cannot meet the zero-second delay (ZSL) requirement for image capture.

[0005] Summary of the Invention

[0006] The present application provides a method for capturing images during video recording, an electronic device, a computer program product, and a computer-readable storage medium, the purpose of which is to achieve image capture that meets ZSL requirements during HDR video recording.

[0007] In order to achieve the above objectives, this application provides the following technical solutions:

[0008] In a first aspect, the present application provides a method for capturing images in a video, comprising: displaying a video recording interface of a camera application, wherein the dynamic range of the image displayed on the video recording interface is greater than a first value, the first value indicating that the image displayed on the video recording interface is a non-low dynamic range image, and the image displayed on the video recording interface is stored in a cache area; in response to a capture instruction in the video recording, the image corresponding to the capture moment in the cache area is used as a captured image, and the capture moment is the moment corresponding to the capture instruction.

[0009] The above information demonstrates that the cache stores the image displayed on the recording interface. This means that images captured by the camera are stored in the cache. This image is larger than the first value, and the first value indicates that the image is not a low-dynamic-range image, indicating that the camera outputs a high-dynamic-range image. The recording interface is an HDR recording interface. When a capture instruction is issued during recording, the image corresponding to the capture moment in the cache is used as the captured image, with the capture moment being the time corresponding to the capture instruction, ensuring the ZSL requirements for image capture.

[0010] In one possible embodiment, the image stored in the buffer is a first raw image and a second raw image captured by the image sensor, obtained through high dynamic range (HDR) fusion and compression processing, where the exposure duration of the first raw image is greater than the exposure duration of the second raw image. In this possible embodiment, compression processing, as opposed to truncation, corresponds to decompression processing, and the compressed image can be restored through decompression, thereby avoiding loss of the image's dynamic range.

[0011] In one possible embodiment, before using the image in the cache corresponding to the capture moment as the captured image, the method further includes: processing the image in the cache corresponding to the capture moment, using the processed image in the cache corresponding to the capture moment as the captured image, wherein the dynamic range of the captured image is greater than a second value, and the second value is greater than the first value. In this way, the captured image can achieve a higher dynamic range.

[0012] In one possible embodiment, processing the image in the cache corresponding to the capture moment includes performing format conversion, decompression, and image processing on the image in the cache corresponding to the capture moment. The format conversion, decompression, and image processing can be understood as being performed sequentially. Due to limitations of format conversion, the image in the cache needs to be compressed. After format conversion, the image is decompressed, and then image processing is performed on the decompressed image to ensure image quality.

[0013] In one possible embodiment, the recording interface displays a first button, and in response to a capture instruction in the recording, the image corresponding to the capture moment in the cache area is used as the capture image, and the capture moment is the moment corresponding to the capture instruction, including: in response to the user's click operation on the first button, the image corresponding to the capture moment in the cache area is used as the capture image, and the capture moment is the moment when the first button is clicked, thereby enabling the user to manually capture images during the recording process.

[0014] In one possible embodiment, in response to a capture instruction in a video, the image corresponding to the capture moment in the cache area is used as a capture image, and the capture moment is the moment corresponding to the capture instruction, including: in response to an instruction to identify a wonderful moment, the image corresponding to the wonderful moment in the cache area is used as a capture image, thereby realizing automatic image capture by the electronic device.

[0015] In one possible implementation, the method is applied to an electronic device, where the electronic device includes a first image sensor and a second image sensor, and the image stored in the buffer area originates from the first image sensor.

[0016] In one possible embodiment, in response to a capture instruction in the video, after using the image corresponding to the capture moment in the cache area as the capture image, it also includes: receiving an operation to switch the camera; in response to the operation to switch the camera, controlling the first image sensor to stop running, and controlling the second image sensor to run, and the image of the second image sensor forms a video recording interface.

[0017] In one possible implementation, controlling the second image sensor to operate includes: controlling the second image sensor to operate in a non-HDR mode, wherein a dynamic range of an image captured by the second image sensor in the non-HDR mode is smaller than a first value.

[0018] In one possible implementation, the method further includes: detecting that the electronic device is in a high-brightness environment, controlling the second image sensor to operate in HDR mode, wherein the dynamic range of the image captured by the second image sensor in HDR mode is greater than the first value. In this way, the switching between HDR mode and non-HDR mode of multiple cameras is achieved.

[0019] In a second aspect, the present application provides an electronic device comprising: one or more processors, a memory, a display screen, and an image sensor; the memory, the display screen, and the image sensor are coupled to one or more processors, the memory is used to store a computer program, the computer program comprises computer instructions, and when one or more processors execute the computer instructions, the electronic device executes the method for capturing an image in a video provided in the first aspect and any one of its possible implementations.

[0020] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the method for capturing images in a video as provided in the first aspect and any one of its possible implementations.

[0021] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method for capturing an image in a video provided in the first aspect and any one of its possible implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a diagram showing an image output by an image sensor provided in the related art;

[0023] FIG2 is a diagram showing an image output by an image sensor according to an embodiment of the present application;

[0024] FIG3 is a software structure diagram of an electronic device provided in an embodiment of the present application;

[0025] FIG4 is an interactive diagram of a method for capturing an image in a video provided by an embodiment of the present application;

[0026] FIG5 is a diagram showing an image processed by a hardware abstraction layer provided in the related art;

[0027] FIG6 is a diagram showing an image processed by a hardware abstraction layer according to an embodiment of the present application;

[0028] FIG7 is a software structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of this application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless the context clearly indicates otherwise.

[0030] References to "some embodiments" and the like in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in some embodiments" and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0031] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0032] Compared to Standard Dynamic Range (SDR) images, High Dynamic Range (HDR) images can display bright and dark areas of the picture with higher quality, providing image details and better reflecting the visual effects of the real environment. Therefore, the larger the dynamic range, the greater the difference between light and dark that can be clearly presented in the image.

[0033] Currently, when recording video with a single camera on an electronic device, the camera can switch between HDR mode and non-HDR mode. As shown in Figure 1, when the camera is in HDR mode, the image sensor in the camera outputs a long-exposure Raw image and a short-exposure Raw image. These long-exposure Raw images and short-exposure Raw images can be fused through HDR to produce an HDR Raw image. When the camera is in non-HDR mode, the image sensor outputs a normal Raw image. The HDR Raw image in HDR mode or the normal Raw image in non-HDR mode can be considered the Raw image of the image sensor. A Raw image can be understood as the image obtained by the image sensor converting the captured light source signal into an electrical signal. The image obtained by fusion of the long-exposure Raw image and the short-exposure Raw image through HDR can also be understood as a Raw image and is therefore called an HDR Raw image.

[0034] The Raw image output by the camera during the video recording process can also be used as a captured image of the video. When the camera is controlled to output the Raw image in HDR mode, the output Raw image is an HDR Raw image. In this way, the camera can capture images with HDR effects during the video recording process. However, whether the user inputs an image capture operation during the video recording process or the electronic device actively captures the image during the video recording process, the electronic device needs to respond to the capture instruction and capture one or more frames of images from the HDR Raw image output by the image sensor to form a captured image. However, since it takes a certain amount of time for the electronic device to respond to the capture instruction, the timestamp of the image captured by the electronic device will lag behind the timestamp of the capture action, which cannot meet the zero-second delay (Zero Shutter Lag, ZSL) requirement of image capture.

[0035] Based on this, the embodiment of the present application provides a method for capturing images during video recording, which can achieve ZSL image capture during HDR video recording. Furthermore, the method for capturing images during video recording provided by the embodiment of the present application also supports switching between HDR mode and non-HDR mode of multiple cameras of electronic devices.

[0036] Figure 2 uses an electronic device with three cameras as an example to illustrate the process of outputting raw images from the image sensors of the three cameras in either HDR or non-HDR mode. The image sensors of the three cameras are respectively referred to as Image Sensor 1, Image Sensor 2, and Image Sensor 3. The fields of view of the three cameras can be the same or different.

[0037] As shown in Figure 2, image sensor 1 can operate in either HDR mode or non-HDR mode. In HDR mode, image sensor 1 outputs long-exposure and short-exposure Raw images, while in non-HDR mode, image sensor 1 outputs normal Raw images. Similarly, image sensor 2 can output long-exposure and short-exposure Raw images in HDR mode, and normal Raw images in non-HDR mode. Image sensor 3 can output long-exposure and short-exposure Raw images in HDR mode, and normal Raw images in non-HDR mode.

[0038] In some embodiments, the electronic device can control only one of image sensor 1, image sensor 2, and image sensor 3 to operate at any one time. For example, in the scenario shown in FIG2 where image sensor 1 is controlled to operate in HDR mode, the long-exposure Raw image and short-exposure Raw image output by image sensor 1 are transmitted to the front-end processing, which performs HDR fusion on the long-exposure Raw image and short-exposure Raw image to generate an HDR Raw image. The HDR Raw image generated by the front-end processing can be cached in a queue for use when capturing images. If image sensor 1 is controlled to switch from HDR mode to non-HDR mode, the HDR Raw image cached in the queue is cleared, and the normal Raw image output by image sensor 1 is transmitted to the front-end processing, which does not process it and directly caches it in the queue. Similarly, if image sensor 1 is controlled to switch from non-HDR mode to HDR mode, the normal Raw image cached in the queue is also cleared, and the HDR Raw image is cached again.

[0039] In other embodiments, the electronic device controls the operation of at least two image sensors, and the electronic device may configure a queue for each of the operating image sensors. The queue configured for the image sensor is used to cache the HDR Raw image or ordinary Raw image of the image sensor.

[0040] In some embodiments, the aforementioned queue can be understood as a buffer area of ​​a memory.

[0041] In some embodiments, the front-end processing method for fusing the long-exposure Raw image and the short-exposure Raw image may be: selecting the highlight area of ​​the short-exposure Raw image and the dark area content of the long-exposure Raw image for fusion.

[0042] From the above content, it can be seen that when the image sensor records in HDR mode, the long-exposure Raw image and the short-exposure Raw image output by the image sensor are HDR-fused through front-end processing to obtain an HDR Raw image. The HDR Raw image can be cached in a queue. In this way, the electronic device can select an HDR Raw image in the queue whose timestamp is the same as or extremely close to the timestamp of the capture behavior to form a captured image, thereby meeting the zero-second delay (Zero Shutter Lag, ZSL) requirement for image capture.

[0043] It should be noted that the four trapezoids shown in Figure 2 do not have any practical meaning. They are intended to illustrate that image sensor 1, image sensor 2, and image sensor 3 output long-exposure Raw images and short-exposure Raw images, or ordinary Raw images at the same time, and to illustrate that image sensor 1, image sensor 2, and image sensor 3 cannot all be in operation at the same time.

[0044] To introduce in detail the method for capturing an image in a video provided by an embodiment of the present application, the following description is made in conjunction with the functional modules in the software structure of the electronic device.

[0045] First, let's discuss the software structure of electronic devices. This can be thought of as the layered architecture of their operating systems. These operating systems run on their hardware components and can include iOS, the open-source Android operating system, and Windows.

[0046] The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0047] FIG3 is a block diagram of the software structure of the electronic device according to an embodiment of the present application.

[0048] A layered architecture divides software into several layers, which communicate with each other through software interfaces. In some embodiments, the Android system is divided into three layers: the application layer, the application framework layer, and the hardware abstraction layer. The three-layer architecture also includes a hardware layer.

[0049] The application layer can include multiple applications. For example, Figure 3 shows two applications: camera and gallery.

[0050] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer shown in FIG3 may include a camera access interface, which is used to provide an API and programming framework for camera applications. For example, as shown in FIG3, the camera access interface includes a camera service.

[0051] The hardware abstraction layer provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer. The camera hardware abstraction layer may provide virtual hardware for the camera, and the camera hardware abstraction layer may include a chip platform and an image engine. The chip platform may be understood as the virtual hardware of an image signal processor (ISP). Exemplarily, as shown in FIG3 , the chip platform includes front-end processing and pre-photographing processing. The front-end processing may be used to implement HDR fusion of long-exposure Raw images and short-exposure Raw images, and pre-photographing processing may be implemented. The image engine is used to switch between multiple camera operations, and may also control the image sensor in the camera to operate in HDR mode or non-HDR mode. In some embodiments, the image engine may also process images captured by the image sensor based on a variety of photographing algorithms. For example, the image engine performs related processing such as denoising on the image through algorithm a, and performs related processing such as adding beauty effects (enlarging eyes, slimming the face, smoothing the skin) to the faces included in the image through algorithm b.

[0052] The following describes the process of how the camera application, camera service, image engine, and image sensor cooperate to complete the method for capturing images in a video provided in an embodiment of the present application.

[0053] As shown in FIG4 , the method for capturing an image in a video provided by an embodiment of the present application includes:

[0054] S401 , image sensor 1 outputs an image in non-HDR mode by default, and saves the Raw image to a queue.

[0055] The image sensor 1 may be configured to operate in a non-HDR mode by default.

[0056] When a user launches the camera app, the camera starts running, and image sensor 1 in the camera captures a raw image. Because image sensor 1 is in non-HDR mode, the raw image captured by image sensor 1 is a non-HDR raw image, typically a low dynamic range (LDR) image. The raw image captured by image sensor 1 is used to form the camera preview interface. The raw image captured by image sensor 1 can also be stored in a queue.

[0057] S402: The camera application receives a user input to start recording.

[0058] The user can input an operation to start recording in the camera preview interface of the camera application. For example, the user can select the recording mode in the camera preview interface and click the capture button to start recording. The camera application receives the operation to start recording input by the user.

[0059] S403: The camera application sends a command to start recording to the camera service.

[0060] When the camera application receives the operation to start recording, it sends a command to start recording to the camera service.

[0061] S404: The camera service sends a video recording start instruction to the image engine.

[0062] When the camera service receives the instruction to start recording, it sends the instruction to start recording to the image engine.

[0063] S405: The image engine monitors the environment in which the electronic device is located.

[0064] When the image engine receives the instruction to start recording, it can determine the mode of the image sensor 1 based on the decision rule, that is, determine whether the image sensor 1 continues to maintain the HDR mode or switches to the HDR mode.

[0065] In some embodiments, the image engine may determine the mode of the image sensor 1 based on the brightness of the environment in which the electronic device is located. Based on this, the image engine monitors the environment in which the electronic device is located.

[0066] S406 : The image engine detects that the electronic device is in a high-brightness environment, and controls the image sensor 1 to switch from the non-HDR mode to the HDR mode.

[0067] The image engine detects that the electronic device is in a high-brightness environment, indicating that image sensor 1 is suitable for HDR mode, and then controls image sensor 1 to switch from non-HDR mode to HDR mode. In some embodiments, the image engine determines that the image captured by image sensor 1 contains a light source reaching a certain brightness, which can generally be considered to be the case when the electronic device is in a high-brightness environment.

[0068] In some embodiments, the image sensor 1 is switched from the non-HDR mode to the HDR mode, and the non-HDR mode Raw images stored in the queue are cleared.

[0069] S407 : Image sensor 1 outputs an image in HDR mode and saves the Raw image in a queue.

[0070] When the image sensor 1 operates in HDR mode, it outputs a long-exposure Raw image and a short-exposure Raw image. The long-exposure Raw image and the short-exposure Raw image are 12 bits. The exposure time of the long-exposure Raw image is longer than the exposure time of the short-exposure Raw image.

[0071] In some embodiments, the long-exposure Raw image and the short-exposure Raw image output by the image sensor 1 may be HDR-fused via front-end processing to obtain an HDR Raw image, which is then stored in a queue.

[0072] S408: The camera application receives a snapshot operation input by the user during the video recording.

[0073] When the camera application is recording, the user can manually input a snapshot operation to capture an image.

[0074] In some embodiments, the user may input a snapshot operation by clicking a capture button on the video recording interface, and the camera application receives the snapshot operation input by the user in the video recording.

[0075] S409: The camera application sends a capture instruction to the camera service.

[0076] When the camera application receives the snapshot operation input by the user in the video, it sends a snapshot instruction to the camera service.

[0077] S410: The camera service sends a capture instruction to the image engine.

[0078] When the camera service receives the snapshot instruction, it sends the snapshot instruction to the image engine.

[0079] S411: The image engine selects the HDR Raw image at the capture moment from the queue.

[0080] When the image engine receives a capture command, it parses the capture timestamp, i.e., the capture moment, from the capture command. The image engine then selects the HDR Raw image at the capture moment from the multiple HDR Raw images stored in the queue.

[0081] In some embodiments, the HDR Raw images stored in the queue are configured with a timestamp to indicate the acquisition time of the long-exposure Raw image and the short-exposure Raw image corresponding to the HDR Raw image. The image engine can select an HDR Raw image in the queue whose timestamp is the same as or very close to the capture timestamp as the HDR Raw image at the capture moment.

[0082] S412: The image engine performs image processing on the HDR Raw image to obtain an image with HDR effect.

[0083] In some embodiments, the image engine performs image processing on the HDR Raw image based on multiple photography algorithms. For example, the image engine performs image processing such as denoising on the image using algorithm a, and performs image beautification effects (enlarging eyes, slimming the face, smoothing the skin) on faces included in the image using algorithm b. The image processed by the image engine is an HDR Raw image. Therefore, the image processed by the image engine has an HDR effect, that is, an image with a high dynamic range.

[0084] S413: The image engine recognizes a wonderful moment and selects an HDR Raw image of the wonderful moment from the queue.

[0085] While the camera application is recording, the electronic device can also automatically capture images. In some embodiments, when the camera application enables the capture function during recording, or operates in a mode that enables automatic capture during recording, the image engine can determine whether the HDR Raw image is a highlight based on the image content and clarity information of the HDR Raw images stored in the queue. If the image engine determines that the HDR Raw image is a highlight, it infers that the timestamp of the HDR Raw image is a highlight moment, and the image engine selects the HDR Raw image of the highlight moment from the queue.

[0086] S414: The image engine performs image processing on the HDR Raw image to obtain an image with HDR effect.

[0087] The image engine obtains the HDR Raw image of the wonderful moment in step S413, and then performs image processing on it to obtain an image with HDR effect. The specific image processing process of the image engine can be found in the content of the aforementioned step S412, and will not be repeated here.

[0088] S415: The camera application receives a lens-changing operation input by the user.

[0089] The lens-changing operation may refer to the operation of controlling an electronic device to switch the camera operation during recording. When a user uses a camera to record a video, the user can adjust the field of view to obtain an image with a larger or smaller shooting range. Based on this, the electronic device can be configured with cameras of various forms, such as wide-angle, ultra-wide-angle, telephoto, and other forms of cameras. The user inputs the operation of adjusting the field of view angle in the recording interface, which may cause the electronic device to switch to a different form of camera operation. The operation of adjusting the field of view angle belongs to the lens-changing operation.

[0090] It should be noted that the electronic device does not need to restart the flow in response to the lens-changing operation input by the user.

[0091] S416: The camera application sends a lens change instruction to the camera service.

[0092] When the camera application receives the lens change operation, it sends a lens change instruction to the camera service.

[0093] S417: The camera service sends a lens change instruction to the image engine.

[0094] When the camera service receives the lens change instruction, it sends the lens change instruction to the image engine.

[0095] S418 : The image engine controls the image sensor 1 to stop operating.

[0096] Exemplarily, the image engine receives a lens switching instruction, and in response to the lens switching instruction, switches from operating image sensor 1 to operating image sensor 2. Based on this, the image engine controls image sensor 1 to stop operating.

[0097] S419 : The image engine controls the image sensor 2 to operate in a non-HDR mode.

[0098] In some embodiments, the image sensor 2 may be configured to operate in a non-HDR mode by default. The image engine controls the image sensor 2 to operate in a non-HDR mode.

[0099] In some other embodiments, the image engine may further control the image sensor 2 to continue operating in the mode of the image sensor 1 , that is, control the image sensor 2 to operate in the HDR mode.

[0100] As the image sensor is adjusted during operation, the raw images saved in the queue are usually cleared to store new raw images.

[0101] S420: Image sensor 2 outputs an image in non-HDR mode and saves the Raw image in a queue.

[0102] In some embodiments, when the image sensor 2 is operating in non-HDR mode, the image engine may monitor whether the electronic device is in a high-brightness environment based on step S405, and switch the image sensor 2 to HDR mode if it is in a high-brightness environment.

[0103] It should be noted that the image engine controls image sensor 2 to operate in HDR mode, and image sensor 2 outputs images in HDR mode, resulting in long-exposure Raw images and short-exposure Raw images. These long-exposure Raw images and short-exposure Raw images output by image sensor 1 are then HDR-fused by front-end processing to produce an HDR Raw image, which is then stored in a queue.

[0104] It should also be noted that while the camera is recording, the user can also input the stop recording operation in the recording interface. The camera application receives the stop recording operation and sends a stop recording instruction to the image engine through the camera service. The image engine can then stop the running image sensor from outputting the image.

[0105] In the related art, as shown in Figure 5, the long-exposure Raw image and short-exposure Raw image output by the image sensor are 12 bits. After the front-end processing performs HDR fusion on them, an 18-bit HDR Raw image can be obtained. Due to the limitation of certain processing modules in the chip platform, such as the format conversion module in the pre-photo processing that can only process a maximum of 14-bit Raw images, the 18-bit HDR Raw image obtained by the front-end processing must be truncated into a 14-bit HDR Raw image. The 14-bit HDR Raw image is then format-converted by the format conversion module and then processed by the ISP processing module to obtain a 14-bit Raw image. The 14-bit Raw image is processed by the image engine calling the photography algorithm, and then converted into an image in YUV format by the format conversion module. The YUV format image is encoded into a final image in JPEG format, and the final image in JPEG format can be provided to the upper-layer application for screen display or storage in the gallery.

[0106] In the above process, the 18-bit HDR Raw image obtained by front-end processing is truncated to a 14-bit HDR Raw image, which will damage the dynamic range of the HDR Raw image and affect the quality of the final image.

[0107] To address this issue, in the method for capturing images in a video provided in an embodiment of the present application, the 18-bit HDR Raw image obtained by front-end processing is not truncated to form a 14-bit HDR Raw image. Instead, the 18-bit HDR Raw image is non-linearly compressed to form a 14-bit HDR Raw image. The 14-bit HDR Raw image obtained by non-linear compression can also be decompressed to form an 18-bit HDR Raw image, thus avoiding loss of dynamic range.

[0108] The following details are explained in conjunction with Figure 6. As shown in Figure 6, the image sensor outputs a 12-bit long-exposure Raw image and a 12-bit short-exposure Raw image. The front-end processing performs HDR fusion on the long-exposure Raw image and the short-exposure Raw image to produce an 18-bit HDR Raw image. The compression module in the front-end processing uses a nonlinear compression curve to compress the 18-bit HDR Raw image to produce a 14-bit HDR Raw image. The 14-bit HDR Raw image obtained by the front-end processing is stored in a queue.

[0109] When a user manually captures an image or an electronic device actively captures an image, the pre-photo processing obtains a 14-bit HDR Raw image from the queue. The pre-photo processing can process 14-bit HDR Raw images. For example, the format conversion module converts the format of the 14-bit HDR Raw image to obtain a 14-bit HDR image. Since the ISP module can support processing 18-bit HDR Raw images, in order to improve image quality, the 14-bit HDR image obtained by the format conversion module can be decompressed by the decompression module to obtain an 18-bit HDR Raw image.

[0110] In some embodiments, the decompression processing method can be: the decompression module obtains the nonlinear compression curve used in the compression processing, and based on the nonlinear compression curve, uses curve interpolation to restore the 14-bit HDR Raw image to an 18-bit HDR Raw image.

[0111] The ISP processing module performs image processing on 18-bit HDR Raw images, such as algorithmic optimization of image noise, brightness, and skin tone, and optimization of parameters such as exposure and color temperature of the shooting scene. The 18-bit HDR Raw images processed by the ISP processing module can be further compressed by the compression module to form 14-bit HDR Raw images, which adapt to the original processing logic of the pre-photography processing module.

[0112] The image engine is configured with a decompression module. After the image engine obtains the 14-bit HDR Raw image output by the pre-photography processing, the decompression module decompresses the 14-bit HDR Raw image to obtain an 18-bit HDR Raw image. In some embodiments, the decompression module in the image engine obtains the nonlinear compression curve used by the compression module in the pre-photography processing, and based on the nonlinear compression curve, uses curve interpolation to restore the 14-bit HDR Raw image to an 18-bit HDR Raw image. The decompression module decompresses the 14-bit HDR Raw image into an 18-bit HDR Raw image, which can provide the image engine with a high dynamic range image and ensure the image quality when the image engine uses the photography algorithm to process the image.

[0113] The image engine uses a camera algorithm to process the 18-bit HDR Raw image, which is then converted to a YUV format through a format conversion module. The YUV image is then encoded into a final image in a JPEG format, which can then be provided to higher-level applications for display or saved in an image library.

[0114] It should be noted that during the research process, the inventors discovered that due to technical limitations, the long-exposure Raw and short-exposure Raw images output by the image sensor in the related art shown in Figure 5 are 12-bit. After the front-end processing performs HDR fusion on these images, an 18-bit HDR Raw image cannot be obtained, and the resulting HDR Raw image is less than 18 bits. Therefore, the HDR Raw images less than 18 bits are truncated to 14-bit HDR Raw images, which limits the loss of dynamic range and is within a tolerable range.

[0115] However, with the continuous development of technology, under the influence of new generation technologies, front-end processing can generate 18-bit HDR Raw images by HDR fusion of long-exposure Raw images and short-exposure Raw images output by image sensors. If the 18-bit HDR Raw images are further truncated to 14-bit HDR Raw images, the loss of dynamic range will exceed the acceptable range. Therefore, it is increasingly urgent to replace this truncation method with other methods.

[0116] The inventors' research has discovered that compression technology can replace truncation technology, enabling the conversion of 18-bit HDR Raw images to 14-bit HDR Raw images. They have also discovered that decompression technology can be used to restore 14-bit HDR Raw images to 18-bit HDR Raw images, avoiding loss of dynamic range. Furthermore, this decompression technology can be applied to image engines. Based on these findings, the solution provided in the embodiments of this application was developed.

[0117] It should also be noted that, as shown in Figure 4, the image sensor can operate in non-HDR mode to obtain low dynamic range images, or it can operate in HDR mode to obtain medium-high dynamic range images. Electronic devices can also obtain high dynamic range images based on the technology shown in Figure 6. In this way, electronic devices can obtain low dynamic range, medium-high dynamic range, and high dynamic range images, and support the acquisition of full dynamic range images (or full scene images). Generally speaking, high dynamic range can refer to medium-high dynamic range and high dynamic range, or non-low dynamic range.

[0118] Electronic devices using the technology shown in Figure 6 can achieve an image dynamic range exceeding 14 bits, significantly greater than 9 bits, and thus qualifying as a high dynamic range image. Generally, an image with a dynamic range exceeding 9 bits is considered to have a high dynamic range.

[0119] The video recording mentioned above can be called video recording in a broad sense, and is not limited to the camera shooting videos in video mode, but can also refer to the camera shooting videos in other modes, such as movie mode.

[0120] The electronic devices disclosed in the embodiments of the present application can be mobile phones, tablet computers, personal digital assistants (PDAs), desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, wearable devices and other electronic devices.

[0121] Taking a mobile phone as an example, Figure 7 shows an example of the composition of an electronic device provided in an embodiment of the present application. As shown in Figure 7, the electronic device 700 may include a processor 710, an internal memory 720, a camera 730, a display screen 740, and a sensor module 750.

[0122] It should be understood that, to facilitate understanding of the embodiments of the present application, the electronic device 100 shown in FIG7 only includes some components related to the method for capturing an image in a video provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application may include more or fewer components than the electronic device 100 shown in FIG7. In other words, the electronic device 100 shown in FIG7 does not constitute a specific limitation on the electronic device provided in the embodiments of the present application.

[0123] The processor 710 may include one or more processing units, such as an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), and a video codec. The processor 710 may also include a memory for storing instructions and data.

[0124] The internal memory 720 can be used to store computer-executable program code, which includes instructions. The processor 710 executes the instructions stored in the internal memory 720 to perform various functional applications and data processing of the electronic device 700. In some embodiments, the internal memory 720 stores instructions for executing a method for capturing images in a video. By executing the instructions stored in the internal memory 720, the processor 710 can capture HDR images in the video that meet ZSL requirements.

[0125] The electronic device 700 can implement a shooting function through the ISP, camera 730, video codec, GPU, display screen 740, and application processor, etc. The electronic device 700 can implement a display function through the GPU, display screen 740, and application processor, etc.

[0126] In sensor module 750, pressure sensor 750A senses pressure signals and converts them into electrical signals. Touch sensor 750B, also known as a "touch control device," can be mounted on display screen 740. Together, touch sensor 750B and display screen 740 form a touch screen, also known as a "touch screen." Touch sensor 750B detects touch operations applied to or near the touch sensor. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event.

[0127] Another embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0128] The computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0129] Another embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

Claims

1. A method for capturing an image in a video, characterized in that, Including: Displaying a video recording interface of a camera application, where the dynamic range of the image displayed in the video recording interface is greater than a first value, the first value indicating that the image is not a low-dynamic-range image, and the image is stored in a buffer; In response to a capture instruction during video recording, using the image in the buffer corresponding to the capture moment as a captured image, the capture moment being the moment corresponding to the capture instruction.

2. The method for capturing an image from a video according to claim 1, wherein The images stored in the buffer are obtained by high-dynamic-range (HDR) fusion and compression processing of a first Raw image and a second Raw image collected by an image sensor, and the exposure duration of the first Raw image is greater than that of the second Raw image.

3. The method for capturing an image from a video according to claim 2, wherein, Before using the image in the buffer corresponding to the capture moment as a captured image, it further includes: Processing the image in the buffer corresponding to the capture moment, and using the processed image in the buffer corresponding to the capture moment as the captured image, where the dynamic range of the captured image is greater than a second value, and the second value is greater than the first value.

4. The method for capturing an image from a video according to claim 3, wherein The processing of the image in the buffer corresponding to the capture moment includes: Performing format conversion, decompression processing, and image processing on the image in the buffer corresponding to the capture moment.

5. The method for capturing an image from a video according to any one of claims 1 to 4, characterized in that, The video recording interface displays a first button, and in response to a capture instruction during video recording, using the image in the buffer corresponding to the capture moment as a captured image, the capture moment being the moment corresponding to the capture instruction, includes: In response to a user's click operation on the first button, using the image in the buffer corresponding to the click moment of the first button as a captured image.

6. The method for capturing an image in a video according to any one of claims 1 to 4, characterized in that, In response to a capture instruction during video recording, using the image in the buffer corresponding to the capture moment as a captured image, the capture moment being the moment corresponding to the capture instruction, includes: In response to an instruction for identifying a wonderful moment, using the image in the buffer corresponding to the wonderful moment as the captured image.

7. The method for capturing an image from a video according to any one of claims 1 to 6, characterized in that, The method is applied to an electronic device, the electronic device including a first image sensor and a second image sensor, and the images stored in the buffer are sourced from the first image sensor.

8. The method for capturing an image from a video according to claim 7, characterized in that, After using the image in the buffer corresponding to the capture moment as a captured image in response to a capture instruction during video recording, it further includes: Receiving an operation to switch cameras; In response to the operation to switch cameras, controlling the first image sensor to stop operating and controlling the second image sensor to operate, and the image of the second image sensor forms the video recording interface.

9. The method for capturing an image from a video according to claim 8, wherein The controlling the second image sensor to operate includes: controlling the second image sensor to operate in a non-HDR mode, and the dynamic range of the image collected by the second image sensor in the non-HDR mode is less than the first value.

10. The method for capturing an image from a video according to claim 9, characterized in that, It further includes: When it is detected that the electronic device is in a high-brightness environment, controlling the second image sensor to operate in an HDR mode, and the dynamic range of the image collected by the second image sensor in the HDR mode is greater than the first value.

11. An electronic device, characterized in that, Including: One or more processors, a memory, a display screen, and an image sensor; The memory, the image sensor, and the display screen are coupled to the one or more processors. The memory is used to store a computer program, and the computer program includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method for capturing an image during video recording as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, For storing a computer program, when the computer program is executed, it is specifically used to implement the method for capturing an image during video recording as described in any one of claims 1 to 10.

13. A computer program product, characterized in that, When the computer program product runs on a computer, it causes the computer to execute the method for capturing an image during video recording as described in any one of claims 1 to 10.

Images