Method for capturing images in a video recording, electronic device and readable medium

Abstract

The application provides a method for capturing an image in a video, an electronic device and a readable medium. The method comprises: displaying a video interface of a camera application, the dynamic range of the image displayed by the video interface being greater than a first value, the first value indicating that the image displayed by the video interface is a non-low dynamic range image, and the image displayed by the video interface being stored in a cache area; and in response to a capture instruction in the video, taking the image corresponding to the capture time in the cache area as the captured image. The cache area stores the image displayed by the video interface, that is, the image captured by the camera is stored in the cache area. The image is greater than the first value, and the first value indicates that the image is a non-low dynamic range image, which means that the camera outputs a high dynamic range image, and the video interface belongs to an HDR video interface. In the case of a capture instruction in the video, the image corresponding to the capture time in the cache area is taken as the captured image, and the capture time is the time corresponding to the capture instruction, which ensures the ZSL requirement of image capture.

Description

Technical Field

[0001] This application relates to the field of photography technology, and in particular to a method for capturing images in video recording, electronic devices, computer program products, and computer-readable storage media. Background Technology

[0002] High Dynamic Range (HDR) images can display the bright and dark areas of a picture with higher quality, provide image detail, and better reflect the visual effects of the real environment.

[0003] Currently, during video recording by a single camera on an electronic device, the camera can switch between HDR and non-HDR modes. The raw image output by the camera during recording can also be used as a snapshot image. When the camera is controlled to output a raw image in HDR mode, the output raw image is an HDR raw image, thus allowing the camera to capture HDR-effect images during recording. However, the images captured by electronic devices cannot meet the zero-shutter lag (ZSL) requirement for image capture. Summary of the Invention

[0004] This application provides a method, electronic device, computer program product, and computer-readable storage medium for capturing images during video recording, with the aim of enabling image capture that meets ZSL requirements during HDR video recording.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, this application provides a method for capturing an image during video recording, comprising: displaying a video recording interface of a camera application, wherein the dynamic range of the image displayed on the video recording interface is greater than a first value, the first value indicating that the image displayed on the video recording interface is a non-low dynamic range image, and the image displayed on the video recording interface is stored in a buffer; and responding to a capture command in the video recording, using the image in the buffer corresponding to the capture time as the captured image, wherein the capture time is the time corresponding to the capture command.

[0007] As can be seen from the above, the buffer stores the images displayed on the recording interface, i.e., the images captured by the camera are stored in the buffer. This image is larger than a first value, and the first value indicates that the image is not a low dynamic range image, meaning the camera outputs a high dynamic range image, and the recording interface is an HDR recording interface. When a capture command is generated during recording, the image in the buffer corresponding to the capture time is used as the captured image. The capture time is the time corresponding to the capture command, ensuring the ZSL requirement for image capture.

[0008] In one possible implementation, the image stored in the buffer is obtained by high dynamic range (HDR) fusion and compression of a first Raw image and a second Raw image acquired by an image sensor, wherein the exposure duration of the first Raw image is longer than that of the second Raw image. In this possible implementation, compression processing, compared to truncation, corresponds to decompression processing, and the compressed image can be restored through decompression processing, thus avoiding damage to the dynamic range of the image.

[0009] In one possible implementation, before using the image in the buffer corresponding to the capture time as the captured image, the method further includes: processing the image in the buffer corresponding to the capture time, and using the processed image in the buffer corresponding to the capture time as the captured image. The dynamic range of the captured image is greater than a second value, and the second value is greater than a first value. In this way, the captured image can achieve a higher dynamic range.

[0010] In one possible implementation, the images in the buffer corresponding to the capture time are processed, including: format conversion, decompression, and image processing. Format conversion, decompression, and image processing can be understood as being performed sequentially. Due to limitations in format conversion, the images in the buffer need to be compressed. After format conversion and decompression, image processing is performed on the decompressed images to ensure image quality.

[0011] In one possible implementation, the recording interface displays a first button that, in response to a capture command in the recording, uses the image in the buffer corresponding to the capture time as the captured image. The capture time is the time corresponding to the capture command. This includes: in response to the user clicking the first button, using the image in the buffer corresponding to the capture time as the captured image. The capture time is the time the first button is clicked, thus enabling the user to manually capture images during the recording process.

[0012] In one possible implementation, in response to a capture command in the video recording, the image in the buffer corresponding to the capture time is used as the captured image, where the capture time is the time corresponding to the capture command. This includes: in response to a command that identifies a highlight moment, the image in the buffer corresponding to the highlight moment is used as the captured image, thus enabling the electronic device to automatically capture images.

[0013] In one possible implementation, the method applies an electronic device, which includes a first image sensor and a second image sensor, and the image stored in the buffer originates from the first image sensor.

[0014] In one possible implementation, after taking the image corresponding to the capture time in the buffer as the captured image in response to the capture command in the video recording, the method further includes: receiving an operation to switch cameras; in response to the operation to switch cameras, controlling the first image sensor to stop operating and controlling the second image sensor to operate, with the image from the second image sensor forming the video recording interface.

[0015] In one possible implementation, controlling the operation of the second image sensor includes: controlling the second image sensor to operate in a non-HDR mode, wherein the dynamic range of the images acquired by the second image sensor in the non-HDR mode is less than a first value.

[0016] In one possible implementation, the method further includes: detecting that the electronic device is in a high-brightness environment, controlling the second image sensor to operate in HDR mode, wherein the dynamic range of the image acquired by the second image sensor in HDR mode is greater than a first value. This enables multiple cameras to switch between HDR and non-HDR modes.

[0017] In a second aspect, this application provides an electronic device comprising: one or more processors, a memory, a display screen, and an image sensor; the memory, the display screen, and the image sensor are coupled to one or more processors, the memory being used to store a computer program, the computer program including computer instructions, and when one or more processors execute the computer instructions, the electronic device performs a method for capturing images in a video recording as provided in any of the first aspects.

[0018] Thirdly, this application provides a computer-readable storage medium for storing a computer program, which, when executed, is specifically used to implement the method for capturing images in a video recording as provided in any of the first aspects.

[0019] Fourthly, this application provides a computer program product that, when run on a computer, causes the computer to execute the method for capturing images in a video recording as provided in any of the first aspects. Attached Figure Description

[0020] Figure 1 A diagram illustrating the image output from an image sensor provided in related technologies;

[0021] Figure 2 This is a diagram illustrating the image output from the image sensor provided in an embodiment of this application.

[0022] Figure 3 A software structure diagram of the electronic device provided in the embodiments of this application;

[0023] Figure 4 An interactive diagram illustrating the method for capturing images from a video recording as provided in this application embodiment;

[0024] Figure 5 This is a diagram illustrating the image processing capabilities of the hardware abstraction layer provided in related technologies.

[0025] Figure 6 An illustration of the hardware abstraction layer processing image provided in an embodiment of this application;

[0026] Figure 7 A software structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise.

[0028] References to "some embodiments" and the like in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in some embodiments," "in other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiments, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0029] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0030] Compared to Standard Dynamic Range (SDR) images, High Dynamic Range (HDR) images can display both bright and dark areas of an image with higher quality, providing more detail and better reflecting the visual effects of the real environment. Therefore, the greater the dynamic range, the more clearly the image can present differences in brightness and darkness.

[0031] Currently, during the recording process of a single camera on an electronic device, the camera can switch between HDR mode and non-HDR mode. For example... Figure 1As shown, in HDR mode, the camera's image sensor outputs long-exposure Raw and short-exposure Raw images. These images can be fused using HDR to obtain an HDR Raw image. In non-HDR mode, the image sensor outputs a regular Raw image. The HDR Raw image in HDR mode or the regular Raw image in non-HDR mode can be used as the image sensor's Raw image. A Raw image can be understood as the image obtained after the image sensor converts the captured light source signal into an electrical signal. The image obtained by HDR fusion of the long-exposure and short-exposure Raw images can also be considered a Raw image; therefore, it is called an HDR Raw image.

[0032] The Raw image output by the camera during recording can also be used as a snapshot image. When the camera is controlled to output Raw image in HDR mode, the output Raw image is an HDR Raw image, thus allowing the camera to capture HDR-effect images during recording. However, whether the user inputs an image capture operation during recording or the electronic device actively captures an image during recording, the electronic device needs to respond to the capture command and extract one or more frames from the HDR Raw image output by the image sensor to form the captured image. However, because there is a certain time delay in the electronic device responding to the capture command, the timestamp of the captured image will lag behind the timestamp of the capture action, failing to meet the zero-short-time (ZSL) requirement for image capture.

[0033] Based on this, embodiments of this application provide a method for capturing images during video recording, enabling ZSL image capture during HDR video recording. Furthermore, the method for capturing images during video recording provided in this application also supports switching between HDR and non-HDR modes using multiple cameras on an electronic device.

[0034] Figure 2 Taking an electronic device comprising three cameras as an example, this demonstrates the process by which the image sensors of the three cameras output Raw images in HDR or non-HDR mode. The image sensors of the three cameras are referred to as Image Sensor 1, Image Sensor 2, and Image Sensor 3. The field of view ranges of the three cameras may be the same or different.

[0035] like Figure 2As shown, image sensor 1 can operate in HDR mode or non-HDR mode. In HDR mode, image sensor 1 outputs long exposure Raw images and short exposure Raw images; in non-HDR mode, image sensor 1 outputs a normal Raw image. Similarly, image sensor 2 can output long exposure Raw images and short exposure Raw images in HDR mode, and a normal Raw image in non-HDR mode; image sensor 3 can output long exposure Raw images and short exposure Raw images in HDR mode, and a normal Raw image in non-HDR mode.

[0036] In some embodiments, the electronic device can control image sensor 1, image sensor 2, and image sensor 3 to operate only one at a time. For example, Figure 2 The image sensor 1 is controlled to operate. In scenes where image sensor 1 is controlled to operate in HDR mode, the long-exposure Raw image and short-exposure Raw image output by image sensor 1 are transmitted to the front-end processing. The front-end processing can perform HDR fusion on the long-exposure Raw image and short-exposure Raw image to obtain an HDR Raw image. The HDR Raw image obtained by the front-end processing can be cached in a queue for use when capturing images. When image sensor 1 is controlled to switch from HDR mode to non-HDR mode, the HDR Raw image cached in the queue is cleared, and the ordinary Raw image output by image sensor 1 is transmitted to the front-end processing. The front-end processing does not process it, and the ordinary Raw image is directly cached in the queue. Similarly, when image sensor 1 is controlled to switch from non-HDR mode to HDR mode, the ordinary Raw image cached in the queue is also cleared, and then the HDR Raw image is cached.

[0037] In other embodiments, the electronic device controls the operation of at least two image sensors, and the electronic device can configure a queue for each operating image sensor. The queue configured for the image sensor is used to cache the HDR Raw image or the normal Raw image of the image sensor.

[0038] In some embodiments, the aforementioned queue can be understood as a cache area in memory.

[0039] In some embodiments, the front-end processing method for fusing long exposure Raw images and short exposure Raw images can be: selecting the highlight areas of the short exposure Raw image and the dark areas of the long exposure Raw image for fusion.

[0040] As can be seen from the above, when the image sensor records video in HDR mode, the long exposure Raw image and short exposure Raw image output by the image sensor are fused by the front-end processing to obtain an HDR Raw image. This HDR Raw image can be cached in a queue. In this way, the electronic device can select an HDR Raw image from the queue whose timestamp is the same as or very close to the timestamp of the capture action to form a capture image, which meets the zero-second delay (Zero Shutter Lag, ZSL) requirement for image capture.

[0041] It should be noted that, Figure 2 The four trapezoids shown are not of practical significance; they are meant to illustrate that image sensor 1, image sensor 2, and image sensor 3 output long exposure Raw and short exposure Raw, or ordinary Raw, at any given moment, and to show that image sensor 1, image sensor 2, and image sensor 3 cannot all be in operation at any given moment.

[0042] To provide a detailed description of the method for capturing images in video recordings provided in this application embodiment, the following explanation will be based on the functional modules in the software structure of the electronic device.

[0043] First, let's introduce the software architecture of electronic devices. The software architecture of an electronic device can be understood as the layered architecture of its operating system. The operating system of an electronic device runs on top of its hardware components and can be, for example, iOS, the Android open-source operating system, or Windows.

[0044] This application uses the layered architecture of the Android system as an example to illustrate the software structure of an electronic device.

[0045] Figure 3 This is a software structure block diagram of an electronic device according to an embodiment of this application.

[0046] A layered architecture divides software into several layers, which communicate with each other through software interfaces. In some embodiments, the Android system is divided into three layers: the application layer, the application framework layer, and the hardware abstraction layer, from top to bottom. Below this three-layer architecture, there is also a hardware layer.

[0047] The application layer can include multiple applications. For example, Figure 3 It showcases two applications: camera and gallery.

[0048] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, Figure 3The demonstrated application framework layer may include a camera access interface, providing application programming interfaces and a programming framework for camera applications. For example, such as... Figure 3 As shown, the camera access interface includes camera services.

[0049] The hardware abstraction layer provides a virtual hardware platform for the operating system. For example, the hardware abstraction layer may include a camera hardware abstraction layer. The camera hardware abstraction layer can provide virtual hardware for the camera, and may include a chip platform and an image engine. The chip platform can be understood as the virtual hardware of an image signal processor (ISP), for example, such as... Figure 3 As shown, the chip platform includes front-end processing and pre-image processing. Front-end processing can be used to perform HDR fusion of long-exposure Raw images and short-exposure Raw images, while pre-image processing can also be implemented. The image engine is used to switch between multiple camera operations and can also control the image sensors in the cameras to operate in HDR mode or non-HDR mode. In some embodiments, the image engine can also process the images acquired by the image sensors based on various image-taking algorithms. For example, the image engine can perform noise reduction and other related processing on the image using algorithm a, and perform beautification effects (such as enlarging eyes, slimming the face, and smoothing skin) on the face in the image using algorithm b.

[0050] The following describes the process by which camera applications, camera services, image engines, and image sensors work together to complete the method for capturing images in video recordings provided in the embodiments of this application.

[0051] like Figure 4 As shown in the embodiments of this application, the method for capturing images in a video recording includes:

[0052] S401, Image Sensor 1 outputs images in non-HDR mode by default and saves the Raw image to the queue.

[0053] Image sensor 1 can be configured to operate in non-HDR mode by default.

[0054] When a user launches the camera application, the camera starts running, and image sensor 1 in the camera captures a raw image. Because image sensor 1 is in non-HDR mode, the raw image captured by image sensor 1 is a non-HDR raw image, typically a low dynamic range (LDR) image. The raw image captured by image sensor 1 is used to form the camera preview interface. The raw image captured by image sensor 1 can also be stored in a queue.

[0055] S402, The camera application receives user input to start recording.

[0056] Users can input the command to start recording in the camera preview interface of the camera app. For example, a user can select the recording mode in the camera preview interface and click the record button to start recording. The camera app receives the user's input to start recording.

[0057] S403, The camera application sends a command to the camera service to start recording.

[0058] When the camera application receives a request to start recording, it sends a command to the camera service to start recording.

[0059] S404, The camera service sends a command to the image engine to start recording.

[0060] When the camera service receives a command to start recording, it sends the command to the image engine to start recording.

[0061] S405, Image Engine monitors the environment in which electronic equipment is located.

[0062] When the image engine receives the instruction to start recording, it can determine the mode of image sensor 1 based on decision rules, that is, whether image sensor 1 should continue to maintain HDR mode or switch to HDR mode.

[0063] In some embodiments, the image engine can determine the mode of the image sensor 1 based on the brightness of the environment in which the electronic device is located. Based on this, the image engine monitors the environment in which the electronic device is located.

[0064] S406 The image engine monitors the electronic device in a high-brightness environment and controls the image sensor 1 to switch from non-HDR mode to HDR mode.

[0065] If the image engine detects that the electronic device is in a high-brightness environment, indicating that image sensor 1 is suitable for HDR mode, it will control image sensor 1 to switch from non-HDR mode to HDR mode. In some embodiments, the image engine determines that there is a light source with a certain brightness in the image acquired by image sensor 1, which is generally considered to indicate that the electronic device is in a high-brightness environment.

[0066] In some embodiments, image sensor 1 switches from non-HDR mode to HDR mode, and the raw images of non-HDR mode stored in the queue are cleared.

[0067] S407, Image Sensor 1 outputs an image in HDR mode and saves the Raw image to the queue.

[0068] When image sensor 1 is operating in HDR mode, it outputs a long exposure Raw image and a short exposure Raw image, both of which are 12-bit. The exposure time of the long exposure Raw image is longer than that of the short exposure Raw image.

[0069] In some embodiments, the long exposure Raw image and short exposure Raw image output by image sensor 1 can be HDR fused by front-end processing to obtain an HDR Raw image, which is then stored in a queue.

[0070] S408, the camera application receives snapshot commands input by the user during video recording.

[0071] During video recording, the camera app allows users to manually input a snapshot command to capture an image.

[0072] In some embodiments, users can input a snapshot operation by clicking the capture button on the recording interface, and the camera application receives the snapshot operation input by the user in the recording.

[0073] S409, The camera application sends a capture command to the camera service.

[0074] When the camera application receives a snapshot command input by the user during video recording, it sends a snapshot instruction to the camera service.

[0075] S410, the camera service sends capture commands to the image engine.

[0076] Once the camera service receives a capture command, it sends the capture command to the image engine.

[0077] S411, The image engine selects the HDR Raw image of the captured moment from the queue.

[0078] Upon receiving a capture command, the image engine can parse the capture timestamp, i.e., the capture time, from the command. The image engine then selects the HDR Raw image corresponding to the capture time from a queue of multiple HDR Raw images stored in the queue.

[0079] In some embodiments, the HDR Raw images stored in the queue are configured with timestamps to indicate the acquisition time of the corresponding long-exposure Raw images and short-exposure Raw images. The image engine can select HDR Raw images in the queue whose timestamps are the same as or very close to the capture timestamps as the HDR Raw images at the capture time.

[0080] S412, the image engine performs image processing on the HDR Raw image to obtain an image with HDR effect.

[0081] In some embodiments, the image engine performs image processing on the HDR Raw image based on various image-taking algorithms. For example, the image engine uses algorithm a to perform noise reduction and other related processing on the image, and uses algorithm b to perform beautification effects (such as enlarging eyes, slimming the face, and smoothing skin) on the face in the image. The image processed by the image engine is an HDR Raw image; therefore, the image after image processing by the image engine has an HDR effect, that is, it is a high dynamic range image.

[0082] S413: The image engine identifies a highlight moment and selects the HDR Raw image of that highlight moment from the queue.

[0083] During video recording, the camera application can also automatically capture images. In some embodiments, the camera application activates the capture function during recording, or runs in a mode that enables automatic capture during recording. The image engine can determine whether an HDR Raw image is impressive based on the image content and sharpness information of HDR Raw images stored in the queue. If the image engine determines that an HDR Raw image is impressive, it infers that the timestamp of that HDR Raw image is a highlight moment, and the image engine selects an HDR Raw image of the highlight moment from the queue.

[0084] S414: The image engine performs image processing on the HDR Raw image to obtain an image with HDR effect.

[0085] The image engine obtains the HDR Raw image of the highlight moment through step S413, and then performs image processing on it to obtain an image with HDR effect. The specific image processing process of the image engine can be found in the aforementioned step S412, and will not be repeated here.

[0086] S415, The camera application receives user input for lens changing operations.

[0087] "Camera switching" refers to the operation of switching cameras during video recording on an electronic device. When recording with a camera, users can adjust the field of view to obtain a wider or narrower image. Based on this, electronic devices can be configured with various camera types, such as wide-angle, ultra-wide-angle, and telephoto lenses. When a user inputs an adjustment to the field of view in the recording interface, the electronic device may switch between different camera types. This adjustment of the field of view constitutes camera switching.

[0088] It should be noted that electronic devices do not need to restart the streaming process when responding to user input for changing the camera.

[0089] S416, The camera application sends a lens-changing command to the camera service.

[0090] When the camera application receives a lens-changing request, it sends a lens-changing instruction to the camera service.

[0091] S417, The camera service sends a lens-changing command to the image engine.

[0092] When the camera service receives a lens-changing instruction, it sends the instruction to the image engine.

[0093] S418, The image engine controls image sensor 1 to stop operating.

[0094] For example, upon receiving a lens-switching command, the image engine switches from operating image sensor 1 to operating image sensor 2 in response to the command. Based on this, the image engine controls image sensor 1 to stop operating.

[0095] S419, the image engine controls image sensor 2 to operate in non-HDR mode.

[0096] In some embodiments, image sensor 2 may be configured to operate in non-HDR mode by default. The image engine controls image sensor 2 to operate in non-HDR mode.

[0097] In other embodiments, the image engine may also control the image sensor 2 to continue operating in the same mode as the image sensor 1, that is, control the image sensor 2 to operate in HDR mode.

[0098] As the image sensor in operation is adjusted, the raw images stored in the queue are usually cleared in order to store new raw images.

[0099] S420 and image sensor 2 output images in non-HDR mode and save the raw images to a queue.

[0100] In some embodiments, while the image sensor 2 is operating in non-HDR mode, the image engine can monitor whether the electronic device is in a high-brightness environment based on step S405. If it is in a high-brightness environment, the image sensor 2 is switched to HDR mode.

[0101] It should be noted that the image engine controls image sensor 2 to operate in HDR mode, and image sensor 2 outputs images in HDR mode, resulting in long exposure Raw images and short exposure Raw images. The long exposure Raw images and short exposure Raw images output by image sensor 1 can be fused in HDR through front-end processing to obtain an HDR Raw image, which is then stored in a queue.

[0102] It should also be noted that during the camera's recording process, the user can also input a stop recording operation on the recording interface. The camera application receives the stop recording operation and sends a stop recording command to the image engine through the camera service. The image engine can then stop the image sensor from outputting the image.

[0103] In related technologies, such as Figure 5As shown, the long-exposure Raw and short-exposure Raw images output by the image sensor are 12-bit. After HDR fusion in the front-end processing, an 18-bit HDR Raw image is obtained. However, due to limitations in certain processing modules within the chip platform, such as the format conversion module in pre-processing, which can only handle a maximum of 14-bit Raw images, the 18-bit HDR Raw image obtained from the front-end processing needs to be truncated to 14-bit. The 14-bit HDR Raw image is then converted by the format conversion module and processed by the ISP module to obtain a 14-bit Raw image. The 14-bit Raw image is then processed by the image engine using the image processing algorithm, and then converted to a format such as YUV. The YUV format image is encoded into a final image such as JPEG, which can then be provided to upper-layer applications for display or storage in a gallery.

[0104] In the above process, the 18-bit HDR Raw image obtained from the front-end processing is truncated to a 14-bit HDR Raw image, which will impair the dynamic range of the HDR Raw image and affect the quality of the final image.

[0105] To address this, in the method for capturing images from a video recording provided in this application embodiment, after the 18-bit HDR Raw image obtained through front-end processing is processed, instead of truncating it to form a 14-bit HDR Raw image, the 18-bit HDR Raw image is non-linearly compressed to form a 14-bit HDR Raw image. The 14-bit HDR Raw image obtained by non-linear compression can also be decompressed into an 18-bit HDR Raw image, thus avoiding loss of dynamic range.

[0106] The following combination Figure 6 Please provide a detailed explanation. For example... Figure 6 As shown, the image sensor outputs a 12-bit long-exposure Raw image and a 12-bit short-exposure Raw image. The front-end processing performs HDR fusion on the long-exposure Raw image and the short-exposure Raw image to obtain an 18-bit HDR Raw image. The compression module in the front-end processing uses a non-linear compression curve to compress the 18-bit HDR Raw image to obtain a 14-bit HDR Raw image. The 14-bit HDR Raw image obtained by the front-end processing is stored in a queue.

[0107] When a user manually captures an image or an electronic device actively captures an image, the pre-processing module retrieves a 14-bit HDR Raw image from the queue. The pre-processing module can process the 14-bit HDR Raw image; for example, the format conversion module converts the 14-bit HDR Raw image to obtain a 14-bit HDR image. Since the ISP module can support processing 18-bit HDR Raw images, to improve image quality, the 14-bit HDR image obtained by the format conversion module can be decompressed by the decompression module to obtain an 18-bit HDR Raw image.

[0108] In some embodiments, the decompression process can be as follows: the decompression module obtains the nonlinear compression curve used in the compression process, and based on the nonlinear compression curve, uses curve interpolation to restore the 14-bit HDR Raw image to an 18-bit HDR Raw image.

[0109] The ISP processing module performs image processing on the 18-bit HDR Raw image, such as optimizing image noise, brightness, and skin tone through algorithms, and optimizing parameters such as exposure and color temperature of the shooting scene. The 18-bit HDR Raw image processed by the ISP processing module can then be compressed by the compression module to form a 14-bit HDR Raw image to adapt to the original processing logic of the pre-processing module.

[0110] The image engine is equipped with a decompression module. After the image engine obtains the 14-bit HDR Raw image output from the pre-processing stage, the decompression module decompresses the 14-bit HDR Raw image to obtain an 18-bit HDR Raw image. In some embodiments, the decompression module in the image engine obtains the non-linear compression curve used by the compression module in the pre-processing stage, and uses curve interpolation based on the non-linear compression curve to restore the 14-bit HDR Raw image to an 18-bit HDR Raw image. The decompression module's decompression of the 14-bit HDR Raw image to an 18-bit HDR Raw image provides the image engine with a high dynamic range image, ensuring image quality when the image engine processes the image using photographic algorithms.

[0111] The image engine uses the image processing algorithm to process the 18-bit HDR Raw image, which is then converted into a YUV format image via a format conversion module. The YUV format image is then encoded into a JPEG format image, which can be provided to upper-layer applications for on-screen display or stored in a gallery.

[0112] It should be noted that the inventors discovered during the research process that, due to technological limitations, Figure 5In the demonstrated technologies, the long-exposure Raw and short-exposure Raw images output by the image sensor are 12-bit. After HDR fusion in the front-end processing, an 18-bit HDR Raw image cannot actually be obtained; the resulting HDR Raw image is smaller than 18 bits. Based on this, the HDR Raw image smaller than 18 bits is truncated to a 14-bit HDR Raw image, and the loss of dynamic range is limited and within a tolerable range.

[0113] However, with the continuous development of technology, under the influence of new-generation technologies, front-end processing can perform HDR fusion of long-exposure Raw and short-exposure Raw images output by the image sensor to obtain an 18-bit HDR Raw image. If the 18-bit HDR Raw image is then truncated to a 14-bit HDR Raw image, the loss of dynamic range will exceed the tolerance range. Based on this, it is increasingly urgent to replace this truncation method with other methods.

[0114] The inventors discovered that compression technology can replace truncation technology, enabling the conversion of an 18-bit HDR Raw image to a 14-bit HDR Raw image. Furthermore, they found that decompression technology can restore a 14-bit HDR Raw image to an 18-bit HDR Raw image, avoiding dynamic range loss. This decompression technology can also be applied to image engines. Based on these findings, the solution provided in this application was developed.

[0115] It should also be noted that, such as Figure 4 As shown, the image sensor can operate in non-HDR mode to obtain low dynamic range images, or in HDR mode to obtain medium-to-high dynamic range images. Electronic devices can also be based on... Figure 6 The demonstrated technology produces high dynamic range (HDR) images, enabling electronic devices to obtain low, medium-high, and high dynamic range (HDR) images, ultimately supporting full dynamic range (or full-scene) images. Generally, high dynamic range can refer to both medium-high and high dynamic range, or non-low dynamic range.

[0116] Electronic devices based on Figure 6 The demonstrated technology produces images with a dynamic range exceeding 14 bits, far greater than 9 bits, thus qualifying as high dynamic range. Generally, an image with a dynamic range exceeding 9 bits is considered to have a high dynamic range.

[0117] The video recordings mentioned above can be referred to in a broad sense, not limited to videos shot by a camera in video recording mode, but also including videos shot by a camera in other modes, such as movie mode.

[0118] The electronic devices disclosed in this application can be mobile phones, tablet computers, personal digital assistants (PDAs), desktop, laptop, and notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices, etc.

[0119] Taking mobile phones as an example, Figure 7 This is an example of the composition of an electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 700 may include a processor 710, an internal memory 720, a camera 730, a display screen 740, and a sensor module 750, etc.

[0120] It should be understood that, for the purpose of facilitating the understanding of the embodiments of this application, Figure 7 The electronic device 100 shown only includes some components related to the method of capturing images in a video recording provided in the embodiments of this application. The electronic device provided in the embodiments of this application may include more than Figure 7 The electronic device shown has 100 or more or fewer components. That is to say, Figure 7 The electronic device 100 shown does not constitute a specific limitation on the electronic device provided in the embodiments of this application.

[0121] Processor 710 may include one or more processing units, such as an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), or a video codec. Processor 710 may also include memory for storing instructions and data.

[0122] Internal memory 720 can be used to store computer executable program code, including instructions. Processor 710 executes various functional applications and data processing of electronic device 700 by running the instructions stored in internal memory 720. In some embodiments, internal memory 720 stores instructions for performing methods of capturing images during video recording. Processor 710 can execute the instructions stored in internal memory 720 to achieve the capture of HDR images that meet ZSL requirements during video recording.

[0123] Electronic device 700 can perform shooting functions through ISP, camera 730, video codec, GPU, display screen 740, and application processor. Electronic device 700 can perform display functions through GPU, display screen 740, and application processor.

[0124] In sensor module 750, pressure sensor 750A is used to sense pressure signals and convert them into electrical signals. Touch sensor 750B, also known as a "touch device," can be placed on display screen 740. The touch sensor 750B and display screen 740 together form a touchscreen, also known as a "touch screen." Touch sensor 750B is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event.

[0125] Another embodiment of this application provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.

[0126] Computer-readable storage media can be non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

[0127] Another embodiment of this application provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.

Claims

1. A method for capturing images during video recording, characterized in that, include: The recording interface of the camera application is displayed. The dynamic range of the image displayed in the recording interface is greater than a first value. The first value indicates that the image is not a low dynamic range image. The image is stored in a buffer area. The image stored in the buffer area is obtained by high dynamic range (HDR) fusion of a first Raw image and a second Raw image acquired by the image sensor. The exposure time of the first Raw image is greater than the exposure time of the second Raw image. In response to a capture command in the video recording, the image in the buffer corresponding to the capture time is used as the capture image, where the capture time is the time corresponding to the capture command.

2. The method for capturing images in a video recording according to claim 1, characterized in that, The first and second Raw images acquired by the image sensor are then fused using high dynamic range (HDR) and compressed.

3. The method for capturing images in a video recording according to claim 2, characterized in that, Before using the image at the corresponding capture time in the buffer as the capture image, the method further includes: The image corresponding to the capture time in the buffer is processed, and the processed image corresponding to the capture time in the buffer is used as the captured image. The dynamic range of the captured image is greater than a second value, and the second value is greater than the first value.

4. The method for capturing images in a video recording according to claim 3, characterized in that, The processing of the image in the buffer corresponding to the capture time includes: The images in the buffer corresponding to the capture time are subjected to format conversion, decompression, and image processing.

5. The method for capturing images in a video recording according to any one of claims 1 to 4, characterized in that, The recording interface displays a first button. In response to a capture command in the recording, the image in the buffer corresponding to the capture time is used as the capture image. The capture time is the time corresponding to the capture command, including: In response to the user's click operation on the first button, the image in the buffer corresponding to the capture time is used as the capture image, and the capture time is the moment when the first button is clicked.

6. The method for capturing images in a video recording according to any one of claims 1 to 4, characterized in that, The method of responding to a capture command in the video recording uses the image in the buffer corresponding to the capture time as the captured image, wherein the capture time is the time corresponding to the capture command, including: In response to the instruction to identify a highlight moment, the image corresponding to the highlight moment in the buffer is used as the captured image.

7. The method for capturing images in a video recording according to any one of claims 1 to 4, characterized in that, The method applies an electronic device, which includes a first image sensor and a second image sensor, and the image stored in the buffer originates from the first image sensor.

8. The method for capturing images in a video recording according to claim 7, characterized in that, After responding to the capture command in the video recording and using the image in the buffer corresponding to the capture time as the captured image, the method further includes: Receive commands to switch cameras; In response to the switching of the camera, the first image sensor is controlled to stop operating, and the second image sensor is controlled to operate, with the image from the second image sensor forming the recording interface.

9. The method for capturing images in a video recording according to claim 8, characterized in that, The control of the second image sensor includes: controlling the second image sensor to operate in a non-HDR mode, wherein the dynamic range of the image acquired by the second image sensor in the non-HDR mode is less than the first value.

10. The method for capturing images in a video recording according to claim 9, characterized in that, Also includes: If the electronic device is detected to be in a high-brightness environment, the second image sensor is controlled to operate in HDR mode, and the dynamic range of the image acquired by the second image sensor in HDR mode is greater than the first value.

11. The method for capturing images in a video recording according to claim 5, characterized in that, The method applies an electronic device, which includes a first image sensor and a second image sensor, and the image stored in the buffer originates from the first image sensor.

12. The method for capturing images in a video recording according to claim 6, characterized in that, The method applies an electronic device, which includes a first image sensor and a second image sensor, and the image stored in the buffer originates from the first image sensor.

13. An electronic device, characterized in that, include: One or more processors, memory, displays, and image sensors; The memory, the image sensor, and the display screen are coupled to the one or more processors. The memory stores a computer program, which includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the method for capturing images in a video recording as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed, is specifically used to implement the method for capturing images in a video recording as described in any one of claims 1 to 12.

Images