Method for capturing image in video recording, electronic equipment and readable medium

Abstract

The invention provides a method for capturing an image in video recording, electronic equipment and a readable medium. The method comprises the steps that a video interface of a camera application is displayed, the dynamic range of an image displayed by the video interface is larger than a first value, the first value indicates that the image displayed by the video interface is a non-low-dynamic-range image, and the image displayed by the video interface is stored in a cache region; and in response to a snapshot instruction in the video, taking an image corresponding to the snapshot moment in the cache region as a snapshot image. The cache region stores the image displayed by the video recording interface, that is, the image collected by the camera is stored in the cache region, the image is greater than the first value, and the first value indicates that the image is a non-low dynamic range image, which indicates that the camera outputs a high dynamic range image, and the video recording interface belongs to an HDR video recording interface. Under the condition that the snapshot instruction is formed in the video, the image corresponding to the snapshot moment in the cache region is taken as the snapshot image, and the snapshot moment is the moment corresponding to the snapshot instruction, so that the ZSL requirement of image snapshot is ensured.

Description

Technical Field

[0001] This application relates to the field of shooting technologies, and in particular, to a method for capturing images during video recording, an electronic device, a computer program product, and a computer-readable storage medium. Background Art

[0002] High Dynamic Range (HDR) images can display the bright and dark parts of a picture with higher quality, can provide image details, and can better reflect the visual effects in the real environment.

[0003] Currently, during the video recording process of a single camera of an electronic device, the camera can switch between HDR mode and non-HDR mode. The Raw image output by the camera during video recording can also be used as a captured image of the video. When the camera is controlled to output a Raw image in HDR mode, the output Raw image is an HDR Raw image. In this way, the camera can capture an image with HDR effect during video recording. However, the images captured by the electronic device cannot meet the zero shutter lag (ZSL) requirement for image capture. Summary of the Invention

[0004] This application provides a method for capturing images during video recording, an electronic device, a computer program product, and a computer-readable storage medium, aiming to achieve image capture that meets the ZSL requirement during HDR video recording.

[0005] To achieve the above objective, this application provides the following technical solutions:

[0006] In a first aspect, this application provides a method for capturing images during video recording, including: displaying a video recording interface of a camera application, where the dynamic range of the image displayed on the video recording interface is greater than a first value, the first value indicating that the image displayed on the video recording interface is a non-low dynamic range image, and the image displayed on the video recording interface is stored in a buffer; in response to a capture instruction during video recording, using the image corresponding to the capture moment in the buffer as the captured image, where the capture moment is the moment corresponding to the capture instruction.

[0007] It can be seen from the above that: the buffer stores the image displayed on the video recording interface, that is, the image captured by the camera will be stored in the buffer. This image is greater than the first value, and the first value indicates that the image is a non-low dynamic range image, indicating that the camera outputs a high dynamic range image, and the video recording interface belongs to the HDR video recording interface. When a capture instruction is formed during video recording, using the image corresponding to the capture moment in the buffer as the captured image, where the capture moment is the moment corresponding to the capture instruction, ensures the ZSL requirement for image capture.

[0008] In a possible implementation, the images stored in the buffer are obtained by performing high-dynamic range (HDR) fusion and compression processing on the first Raw image and the second Raw image collected by an image sensor. The exposure duration of the first Raw image is greater than that of the second Raw image. In this possible implementation, compression processing, as opposed to truncation, can avoid damage to the dynamic range of the image because it corresponds to decompression processing and the compressed image can be restored through decompression processing.

[0009] In a possible implementation, before using the image corresponding to the capture moment in the buffer as the captured image, it further includes: processing the image corresponding to the capture moment in the buffer, and using the processed image corresponding to the capture moment in the buffer as the captured image. The dynamic range of the captured image is greater than a second value, and the second value is greater than a first value. In this way, a higher dynamic range can be achieved for the captured image.

[0010] In a possible implementation, processing the image corresponding to the capture moment in the buffer includes: performing format conversion, decompression processing, and image processing on the image corresponding to the capture moment in the buffer. Format conversion, decompression processing, and image processing can be understood as being executed sequentially. Among them, due to the limitations of format conversion, the images in the buffer need to undergo compression processing. After format conversion and then decompression processing of the image, performing image processing on the decompressed image can ensure the quality of the image.

[0011] In a possible implementation, a first button is displayed on the video recording interface. In response to a capture instruction during video recording, using the image corresponding to the capture moment in the buffer as the captured image, where the capture moment is the moment corresponding to the capture instruction, includes: in response to a user's click operation on the first button, using the image corresponding to the capture moment in the buffer as the captured image, and the capture moment is the moment of clicking the first button. In this way, the user can manually capture images during video recording.

[0012] In a possible implementation, in response to a capture instruction during video recording, using the image corresponding to the capture moment in the buffer as the captured image, where the capture moment is the moment corresponding to the capture instruction, includes: in response to an instruction to identify a wonderful moment, using the image corresponding to the wonderful moment in the buffer as the captured image. In this way, the electronic device can automatically capture images.

[0013] In a possible implementation, the method is applied to an electronic device, which includes a first image sensor and a second image sensor, and the images stored in the buffer are sourced from the first image sensor.

[0014] In a possible implementation, after taking the image corresponding to the capture moment in the buffer as the captured image in response to the capture instruction in the video recording, the following steps are further included: receiving an operation to switch the camera; in response to the operation to switch the camera, controlling the first image sensor to stop running and controlling the second image sensor to run, and the image of the second image sensor forms a video recording interface.

[0015] In a possible implementation, controlling the second image sensor to run includes: controlling the second image sensor to run in a non-HDR mode, and the dynamic range of the image captured by the second image sensor in the non-HDR mode is less than a first value.

[0016] In a possible implementation, the following is further included: when it is detected that the electronic device is in a high-brightness environment, controlling the second image sensor to run in an HDR mode, and the dynamic range of the image captured by the second image sensor in the HDR mode is greater than the first value. In this way, the switching between the HDR mode and the non-HDR mode of multiple cameras is realized.

[0017] In a second aspect, the present application provides an electronic device, including: one or more processors, a memory, a display screen, and an image sensor; the memory, the display screen, and the image sensor are coupled to the one or more processors, and the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method for capturing an image during video recording provided in any item of the first aspect.

[0018] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program, which is specifically used to implement the method for capturing an image during video recording provided in any item of the first aspect when the computer program is executed.

[0019] In a fourth aspect, the present application provides a computer program product, which causes a computer to execute the method for capturing an image during video recording provided in any item of the first aspect when the computer program product runs on the computer. Description of the Drawings

[0020] Figure 1 It is a display diagram of the image output by the image sensor provided in the related art;

[0021] Figure 2 It is a display diagram of the image output by the image sensor provided in the embodiment of the present application;

[0022] Figure 3 It is a software structure diagram of the electronic device provided in the embodiment of the present application;

[0023] Figure 4 It is an interaction diagram of the method for capturing an image during video recording provided in the embodiment of the present application;

[0024] Figure 5 A display diagram of an image processed by a hardware abstraction layer provided in the related art;

[0025] Figure 6 A display diagram of an image processed by a hardware abstraction layer provided in an embodiment of the present application;

[0026] Figure 7 A software structure diagram of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "an", "the", "above-mentioned", "said", and "this" are also intended to include the forms such as "one or more", unless there is a clear opposite indication in the context.

[0028] The reference to "some embodiments" described in this specification means that in one or more embodiments of the present application, specific features, structures, or characteristics described in conjunction with the embodiment are included. Thus, the statements "in some embodiments" and "in other embodiments" that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the terms "first", "second", etc. are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0030] Compared with a Standard Dynamic Range (SDR) image, a High Dynamic Range (HDR) image can display the bright and dark parts of a picture with higher quality, can provide image details, and can better reflect the visual effect in the real environment. Therefore, the larger the dynamic range, the greater the difference in brightness and darkness that the image can clearly present.

[0031] Currently, during the video recording process of a single camera of an electronic device, the camera can switch between the HDR mode and the non-HDR mode. For example Figure 1As shown, the camera is in HDR mode. The image sensor in the camera outputs a long-exposure Raw image and a short-exposure Raw image. The long-exposure Raw image and the short-exposure Raw image can be fused through HDR to obtain an HDR Raw image. When the camera is in non-HDR mode, the image sensor outputs a normal Raw image. The HDR Raw image of the camera in HDR mode or the normal Raw image of the camera in non-HDR mode can be used as the Raw image of the image sensor. The Raw image can be understood as the image obtained after the image sensor converts the captured light source signal into an electrical signal. The image obtained by fusing the long-exposure Raw image and the short-exposure Raw image through HDR can also be understood as belonging to the Raw image. Therefore, it is called an HDR Raw image.

[0032] The Raw image output by the camera during video recording can also be used as a captured image for video recording. When the camera is controlled to output a Raw image in HDR mode, the output Raw image is an HDR Raw image. In this way, the camera can capture an image with HDR effect during video recording. However, whether the user inputs an image capture operation during video recording or the electronic device performs an active capture during video recording, the electronic device needs to respond to the capture instruction and extract one or more frames of images from the HDR Raw image output by the image sensor to form a captured image. However, due to the certain time consumption for the electronic device to respond to the capture instruction, the timestamp of the image captured by the electronic device lags behind the timestamp of the capture behavior, and it cannot meet the zero shutter lag (ZSL) requirement for image capture.

[0033] Based on this, the embodiments of the present application provide a method for capturing an image during video recording, which can achieve ZSL image capture during HDR video recording. Further, the method for capturing an image during video recording provided by the embodiments of the present application also supports the switching between HDR mode and non-HDR mode for multiple cameras of the electronic device.

[0034] Figure 2 Taking an electronic device including three cameras as an example, the process of the image sensors of the three cameras outputting Raw images in HDR mode or non-HDR mode is shown. Among them, the image sensors of the three cameras are respectively called image sensor 1, image sensor 2, and image sensor 3. The field of view angle ranges of the three cameras can be the same or different.

[0035] As Figure 2As shown, the image sensor 1 can operate in HDR mode or non-HDR mode. In HDR mode, the image sensor 1 outputs a long-exposure Raw image and a short-exposure Raw image. In non-HDR mode, the image sensor 1 outputs a normal Raw image. Similarly, the image sensor 2 can output a long-exposure Raw image and a short-exposure Raw image in HDR mode, and can output a normal Raw image in non-HDR mode; the image sensor 3 can output a long-exposure Raw image and a short-exposure Raw image in HDR mode, and can output a normal Raw image in non-HDR mode.

[0036] In some embodiments, the electronic device can control only one of the image sensor 1, the image sensor 2, and the image sensor 3 to operate at the same time. Exemplarily, Figure 2 When the shown image sensor 1 is controlled to operate, in the scenario where the image sensor 1 is controlled to operate in HDR mode, the long-exposure Raw image and the short-exposure Raw image output by the image sensor 1 will be transmitted to the front-end processing. The front-end processing can perform HDR fusion on the long-exposure Raw image and the short-exposure Raw image to obtain an HDR Raw image. The HDR Raw image obtained by the front-end processing can be cached in a queue for use when capturing images. When the image sensor 1 is controlled to switch from HDR mode to non-HDR mode, the HDR Raw image cached in the queue will be cleared, and the normal Raw image output by the image sensor 1 will be transmitted to the front-end processing. The front-end processing does not process it, and the normal Raw image is directly cached in the queue. Similarly, when the image sensor 1 is controlled to switch from non-HDR mode to HDR mode, the normal Raw image cached in the queue is also cleared, and then the HDR Raw image is cached.

[0037] In some other embodiments, the electronic device controls at least two image sensors to operate. The electronic device can respectively configure a queue for each of the operating image sensors, and the queue configured for the image sensor is used to cache the HDR Raw image or the normal Raw image of the image sensor.

[0038] In some embodiments, the aforementioned queue can be understood as a buffer area of the memory.

[0039] In some embodiments, the way for the front-end processing to fuse the long-exposure Raw image and the short-exposure Raw image can be: select the highlight area of the short-exposure Raw image and the dark area content of the long-exposure Raw image for fusion.

[0040] As can be seen from the above: when the image sensor records video in HDR mode, the long-exposure Raw image and short-exposure Raw image output by the image sensor are subjected to HDR fusion through front-end processing to obtain an HDR Raw image, which can be cached in a queue. In this way, the electronic device can select an HDR Raw image with a timestamp the same as or extremely close to the timestamp of the capture behavior from the queue to form a captured image, meeting the zero shutter lag (ZSL) requirement for image capture.

[0041] It should be noted that Figure 2 The four trapezoids shown do not have any substantial meaning. They are used to illustrate that the image sensor 1, image sensor 2, and image sensor 3 output long-exposure Raw images and short-exposure Raw images, or ordinary Raw images, at a certain moment, and to illustrate that the image sensor 1, image sensor 2, and image sensor 3 cannot all be in an operating state at a certain moment.

[0042] To introduce in detail the method for capturing images during video recording provided in the embodiments of the present application, the following will be described in conjunction with the functional modules in the software structure of the electronic device.

[0043] First, introduce the software structure of the electronic device. The software structure of the electronic device can be understood as a layered architecture of the operating system of the electronic device. The operating system of the electronic device runs on the hardware components of the electronic device and can be: iOS operating system, Android open-source operating system, Windows operating system, etc.

[0044] In the embodiments of the present application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device.

[0045] Figure 3 It is the software structure block diagram of the electronic device in the embodiments of the present application.

[0046] The layered architecture divides the software into several layers, and the layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into three layers, from top to bottom: the application layer, the application framework layer, and the hardware abstraction layer. Below the three-layer architecture, there is also a hardware layer.

[0047] The application layer may include multiple applications. Exemplarily, Figure 3 shows two applications: the camera and the gallery.

[0048] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. Exemplarily, Figure 3The displayed application framework layer may include a camera access interface for providing application programming interfaces and programming frameworks for camera applications. Exemplarily, as Figure 3 shown, the camera access interface includes a camera service.

[0049] The hardware abstraction layer provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer. The camera hardware abstraction layer may provide virtual hardware for the camera. The camera hardware abstraction layer may include a chip platform and an image engine. The chip platform may be understood as the virtual hardware of an image signal processor (ISP). Exemplarily, as Figure 3 shown, the chip platform includes front-end processing and pre-photograph processing. The front-end processing can be used to perform HDR fusion on long-exposure Raw images and short-exposure Raw images. The pre-photograph processing can be implemented. The image engine is used to switch the operation of multiple cameras and can also control the image sensor in the camera to operate in HDR mode or non-HDR mode. In some embodiments, the image engine can also process the images collected by the image sensor based on multiple photographing algorithms. For example, the image engine performs relevant processing such as denoising on the image through algorithm a, and performs relevant processing such as adding beauty effects (big eyes, face slimming, skin smoothing) to the face included in the image through algorithm b.

[0050] The following introduces the process of how the camera application, camera service, image engine, and image sensor cooperate to complete the method of capturing an image during video recording provided by the embodiments of the present application.

[0051] As Figure 4 shown, the method of capturing an image during video recording provided by the embodiments of the present application includes:

[0052] S401. The image sensor 1 defaults to outputting images in non-HDR mode and saves the Raw images to the queue.

[0053] The image sensor 1 can be configured to default to operating in non-HDR mode.

[0054] When the user starts the camera application, the camera application starts the camera to operate, and the image sensor 1 in the camera captures Raw images. Since the image sensor 1 is in non-HDR mode, the Raw images captured by the image sensor 1 are non-HDR Raw images, usually low dynamic range (LDR) images. The Raw images captured by the image sensor 1 are used to form the camera preview interface. The Raw images captured by the image sensor 1 can also be stored in the queue.

[0055] S402. The camera application receives the operation of the user to start video recording.

[0056] The user can input an operation to start video recording on the camera preview interface of the camera application. Exemplarily, the user can select the video recording mode on the camera preview interface and click the capture button to start video recording. The camera application receives the operation input by the user to start video recording.

[0057] S403. The camera application sends an instruction to start video recording to the camera service.

[0058] When the camera application receives the operation to start video recording, it sends an instruction to start video recording to the camera service.

[0059] S404. The camera service sends an instruction to start video recording to the image engine.

[0060] When the camera service receives the instruction to start video recording, it sends an instruction to start video recording to the image engine.

[0061] S405. The image engine monitors the environment where the electronic device is located.

[0062] When the image engine receives the instruction to start video recording, it can determine the mode of image sensor 1 based on the decision rule, that is, determine whether image sensor 1 continues to maintain the HDR mode or switches to the HDR mode.

[0063] In some embodiments, the image engine can determine the mode of image sensor 1 based on the brightness of the environment where the electronic device is located. Based on this, the image engine monitors the environment where the electronic device is located.

[0064] S406. When the image engine monitors that the electronic device is in a high-brightness environment, it controls image sensor 1 to switch from the non-HDR mode to the HDR mode.

[0065] When the image engine monitors that the electronic device is in a high-brightness environment, it indicates that image sensor 1 is suitable for the HDR mode, and then controls image sensor 1 to switch from the non-HDR mode to the HDR mode. In some embodiments, when the image engine determines that there is a light source with a certain brightness in the image captured by image sensor 1, it is generally considered that the electronic device is in a high-brightness environment.

[0066] In some embodiments, when image sensor 1 switches from the non-HDR mode to the HDR mode, the Raw images in the non-HDR mode stored in the queue are cleared.

[0067] S407. Image sensor 1 outputs images in the HDR mode and saves the Raw images in the queue.

[0068] When image sensor 1 operates in the HDR mode, it outputs long-exposure Raw images and short-exposure Raw images, and the long-exposure Raw images and short-exposure Raw images are 12-bit. The exposure duration of the long-exposure Raw images is greater than that of the short-exposure Raw images.

[0069] In some embodiments, the long-exposure Raw image and the short-exposure Raw image output by the image sensor 1 can be subjected to HDR fusion through front-end processing to obtain an HDR Raw image, and the HDR Raw image is then saved in the queue.

[0070] S408. The camera application receives the capture operation input by the user during video recording.

[0071] During video recording, the user can manually input a capture operation to capture an image.

[0072] In some embodiments, the user can input a capture operation by clicking the capture button on the video recording interface, and the camera application then receives the capture operation input by the user during video recording.

[0073] S409. The camera application sends a capture instruction to the camera service.

[0074] When the camera application receives the capture operation input by the user during video recording, it sends a capture instruction to the camera service.

[0075] S410. The camera service sends a capture instruction to the image engine.

[0076] When the camera service receives the capture instruction, it sends a capture instruction to the image engine.

[0077] S411. The image engine selects the HDR Raw image at the capture moment from the queue.

[0078] When the image engine receives the capture instruction, it can parse the capture timestamp from the capture instruction, that is, the capture moment. The image engine filters out the HDR Raw image at the capture moment from the multiple frames of HDR Raw images stored in the queue.

[0079] In some embodiments, the HDR Raw images stored in the queue are configured with timestamps to indicate the acquisition moments of the long-exposure Raw images and the short-exposure Raw images corresponding to the HDR Raw images. The image engine can select the HDR Raw image in the queue with a timestamp the same as or very close to the capture timestamp as the HDR Raw image at the capture moment.

[0080] S412. The image engine performs image processing on the HDR Raw image to obtain an image with an HDR effect.

[0081] In some embodiments, the image engine performs image processing on the HDR Raw image based on multiple shooting algorithms. For example, the image engine performs relevant denoising processing on the image through algorithm a, and performs relevant processing such as adding beauty effects (enlarged eyes, thinner face, skin smoothing) to the face included in the image through algorithm b. Since the image processed by the image engine is an HDR Raw image, the image after the image engine performs image processing thus has an HDR effect, that is, it belongs to a high-dynamic-range image.

[0082] S413. The image engine recognizes a wonderful moment and selects the HDR Raw image of the wonderful moment from the queue.

[0083] During video recording by the camera application, the electronic device can also automatically capture images. In some embodiments, the camera application starts the function of capturing images during video recording, or runs in a mode that can achieve automatic image capture during video recording. The image engine can determine whether the HDR Raw image is wonderful based on the image content and clarity information of the HDR Raw images stored in the queue. If the image engine determines that the HDR Raw image is wonderful, it infers that the timestamp of the HDR Raw image is the wonderful moment, and the image engine selects the HDR Raw image of the wonderful moment from the queue.

[0084] S414. The image engine performs image processing on the HDR Raw image to obtain an image with an HDR effect.

[0085] If the image engine obtains the HDR Raw image of the wonderful moment through step S413, it performs image processing on it to obtain an image with an HDR effect. For the specific image processing process of the image engine, refer to the content of step S412 above, which will not be elaborated here.

[0086] S415. The camera application receives the lens-switching operation input by the user.

[0087] The lens-switching operation may refer to an operation that controls the electronic device to switch the operation of the camera during video recording. During the process of the user using the camera to record video, the field of view angle can be adjusted to obtain an image with a larger or smaller shooting range. Based on this, the electronic device can be configured with multiple forms of cameras, such as cameras with wide-angle, ultra-wide-angle, telephoto and other forms. When the user inputs an operation to adjust the field of view angle in the video recording interface, it may cause the electronic device to switch the operation of different forms of cameras. This operation of adjusting the field of view angle belongs to the lens-switching operation.

[0088] It should be noted that when the electronic device responds to the lens-switching operation input by the user, it does not need to restart the stream.

[0089] S416. The camera application sends a lens-switching instruction to the camera service.

[0090] When the camera application receives the lens-switching operation, it sends a lens-switching instruction to the camera service.

[0091] S417. The camera service sends a lens change instruction to the image engine.

[0092] When the camera service receives the lens change instruction, it sends the lens change instruction to the image engine.

[0093] S418. The image engine controls the image sensor 1 to stop running.

[0094] Exemplarily, when the image engine receives the lens change instruction, in response to the lens change instruction, the operation of the image sensor 1 is switched to the operation of the image sensor 2. Based on this, the image engine controls the image sensor 1 to stop running.

[0095] S419. The image engine controls the image sensor 2 to operate in a non-HDR mode.

[0096] In some embodiments, the image sensor 2 may be configured to default to operating in a non-HDR mode. The image engine controls the image sensor 2 to operate in a non-HDR mode.

[0097] In some other embodiments, the image engine may also control the image sensor 2 to continue operating in the mode of the image sensor 1, that is, control the image sensor 2 to operate in an HDR mode.

[0098] Since the operating image sensor is adjusted, the Raw images saved in the queue are usually cleared to store new Raw images.

[0099] S420. The image sensor 2 outputs images in a non-HDR mode and saves the Raw images in the queue.

[0100] In some embodiments, during the process of the image sensor 2 operating in a non-HDR mode, the image engine may monitor whether the electronic device is in a high-brightness environment based on step S405. If it is in a high-brightness environment, the image sensor 2 is switched to the HDR mode.

[0101] It should be noted that when the image engine controls the image sensor 2 to operate in an HDR mode, the image sensor 2 outputs images in an HDR mode to obtain long-exposure Raw images and short-exposure Raw images. The long-exposure Raw images and short-exposure Raw images output by the image sensor 1 can be subjected to HDR fusion through front-end processing to obtain HDR Raw images, and the HDR Raw images are saved in the queue.

[0102] It should also be noted that during the process of the camera recording video, the user can also input a stop video recording operation on the video recording interface. The camera application receives the stop video recording operation, sends a stop video recording instruction to the image engine through the camera service, and the image engine can then stop the image sensor that is running from outputting images.

[0103] In the related art, such as Figure 5As shown in the figure, the long-exposure Raw image and the short-exposure Raw image output by the image sensor are 12 bits. After the front-end processing performs HDR fusion on them, an 18-bit HDR Raw image can be obtained. Limited by some processing modules in the chip platform, such as the format conversion module in the pre-photograph processing can only process a maximum of 14-bit Raw images at most, the 18-bit HDR Raw image obtained by the front-end processing needs to be truncated to a 14-bit HDR Raw image. The 14-bit HDR Raw image is then subjected to format conversion by the format conversion module and then processed by the ISP processing module to obtain a 14-bit Raw image. The 14-bit Raw image is processed by the image engine calling the photographing algorithm for image processing and then converted into an image in a format such as YUV by the format conversion module. The image in YUV format is encoded into a final image in a format such as JPEG, and the final image in JPEG format can be provided to the upper-layer application for screen display or stored in the gallery.

[0104] In the above process, the 18-bit HDR Raw image obtained by the front-end processing is truncated to a 14-bit HDR Raw image, which will cause the dynamic range of the HDR Raw image to be damaged and affect the quality of the final image.

[0105] In view of this, in the method for capturing an image during video recording provided by the embodiment of the present application, after the 18-bit HDR Raw image obtained by the front-end processing, instead of using the truncation method to form a 14-bit HDR Raw image, the 18-bit HDR Raw image is formed into a 14-bit HDR Raw image by using a non-linear compression method. The 14-bit HDR Raw image obtained by using the non-linear compression method can also be decompressed into an 18-bit HDR Raw image, which can avoid damage to the dynamic range.

[0106] The following will be described in detail in conjunction with Figure 6 As shown in Figure 6 the figure, the image sensor outputs a 12-bit long-exposure Raw image and a 12-bit short-exposure Raw image. The front-end processing performs HDR fusion on the long-exposure Raw image and the short-exposure Raw image to obtain an 18-bit HDR Raw image. The compression module in the front-end processing uses a non-linear compression curve to perform compression processing on the 18-bit HDR Raw image to obtain a 14-bit HDR Raw image. The 14-bit HDR Raw image obtained by the front-end processing is stored in a queue.

[0107] When a user manually captures an image or an electronic device actively captures an image, the pre - capture processing obtains a 14 - bit HDR Raw image from the queue. The pre - capture processing can process the 14 - bit HDR Raw image. Exemplarily, the format conversion module performs format conversion on the 14 - bit HDR Raw image to obtain a 14 - bit HDR image. Since the ISP module can support processing 18 - bit HDR Raw images, in order to improve the image quality, the 14 - bit HDR image obtained by the format conversion module can be decompressed by the decompression module to obtain an 18 - bit HDR Raw image.

[0108] In some embodiments, the way of decompression processing can be: the decompression module obtains the non - linear compression curve used in the compression processing, and based on the non - linear compression curve, uses curve interpolation to restore the 14 - bit HDR Raw image to an 18 - bit HDR Raw image.

[0109] The ISP processing module performs image processing on the 18 - bit HDR Raw image, such as optimizing algorithms for image noise, brightness, and skin color, and optimizing parameters such as exposure and color temperature of the shooting scene. The 18 - bit HDR Raw image processed by the ISP processing module can be compressed by the compression module to form a 14 - bit HDR Raw image to adapt to the original processing logic of the pre - capture processing module.

[0110] The image engine is configured with a decompression module. After the image engine obtains the 14 - bit HDR Raw image output by the pre - capture processing, the decompression module decompresses the 14 - bit HDR Raw image to obtain an 18 - bit HDR Raw image. In some embodiments, the decompression module in the image engine obtains the non - linear compression curve used by the compression module in the pre - capture processing, and based on the non - linear compression curve, uses curve interpolation to restore the 14 - bit HDR Raw image to an 18 - bit HDR Raw image. Decompressing the 14 - bit HDR Raw image into an 18 - bit HDR Raw image can provide a high - dynamic - range image for the image engine and ensure the image quality when the image engine processes the image using the shooting algorithm.

[0111] The image engine calls the shooting algorithm to perform image processing on the 18 - bit HDR Raw image, and then converts it into an image in a format such as YUV through the format conversion module. The image in YUV format is encoded into a final image in a format such as JPEG, and the final image in JPEG format can be provided to the upper - layer application for screen display or saved in the photo library.

[0112] It should be noted that the inventors found during the research of the solution that: limited by technological limitations, Figure 5In the related art shown, the long-exposure Raw image and the short-exposure Raw image output by the image sensor are 12 bits. After the front-end processing performs HDR fusion on them, in essence, an 18-bit HDR Raw image cannot be obtained, and the obtained HDR Raw image is less than 18 bits. Based on this, the HDR Raw image less than 18 bits is truncated to a 14-bit HDR Raw image, and the loss of dynamic range is limited and within the tolerable range.

[0113] However, with the continuous development of technology, under the influence of the new generation of technology, the front-end processing performs HDR fusion on the long-exposure Raw image and the short-exposure Raw image output by the image sensor, and an 18-bit HDR Raw image can be obtained. If the 18-bit HDR Raw image is truncated to a 14-bit HDR Raw image again, the loss of dynamic range will exceed the tolerable range. Based on this, this truncation method increasingly urgently needs to be replaced by other methods.

[0114] Through the research of the inventor, it is found that the compression technology can replace the truncation technology and can realize adjusting the 18-bit HDR Raw image to a 14-bit HDR Raw image. It is also found that the 14-bit HDR Raw image can be restored to an 18-bit HDR Raw image by using the decompression technology, avoiding the loss of dynamic range. Moreover, this decompression technology can also be applied to the image engine. Based on this research finding, the solution provided in the embodiments of this application emerges as the times require.

[0115] It should also be noted that as Figure 4 shown, the image sensor can operate in the non-HDR mode to obtain an image with a low dynamic range, or can also operate in the HDR mode to obtain an image with a medium-high dynamic range. The electronic device can also obtain an image with a high dynamic range based on Figure 6 the technology shown. In this way, the electronic device can obtain images with a low dynamic range, a medium-high dynamic range, and a high dynamic range, realizing support for obtaining images with a full dynamic range (or full-scene images). Usually, the high dynamic range can generally refer to the medium-high dynamic range and the high dynamic range, or be called the non-low dynamic range.

[0116] The electronic device based on Figure 6 the technology shown can obtain an image with a dynamic range reaching more than 14 bits, which is much larger than 9 bits and can belong to the high dynamic range. Usually, when the dynamic range of an image exceeds 9 bits, it can be determined that the image belongs to the high dynamic range image.

[0117] The video recording proposed in the foregoing content can be called generalized video recording, which is not limited to the camera shooting video in the video recording mode, and can also refer to the camera shooting video in other modes, such as the movie mode, etc.

[0118] The electronic device disclosed in the embodiments of the present application may be an electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a desktop computer, a laptop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, and a wearable device.

[0119] Taking a mobile phone as an example, Figure 7 This is a component example of an electronic device provided by the embodiments of the present application. As Figure 7 shown, the electronic device 700 may include a processor 710, an internal memory 720, a camera 730, a display screen 740, a sensor module 750, etc.

[0120] It should be understood that for the convenience of understanding the embodiments of the present application, Figure 7 the electronic device 100 shown only includes some components related to the method of capturing images during video recording provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application may include more or fewer components than Figure 7 the electronic device 100 shown. That is to say, Figure 7 the electronic device 100 shown does not constitute a specific limitation on the electronic device provided by the embodiments of the present application.

[0121] The processor 710 may include one or more processing units. For example, the processor 710 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), and a video codec. A memory may also be provided in the processor 710 for storing instructions and data.

[0122] The internal memory 720 may be used to store computer-executable program code, and the executable program code includes instructions. The processor 710 executes various functional applications and data processing of the electronic device 700 by running the instructions stored in the internal memory 720. In some embodiments, the instructions stored in the internal memory 720 are for executing the method of capturing images during video recording. The processor 710 may implement capturing HDR images that meet the ZSL requirements during video recording by executing the instructions stored in the internal memory 720.

[0123] The electronic device 700 may implement the shooting function through the ISP, the camera 730, the video codec, the GPU, the display screen 740, and the application processor, etc. The electronic device 700 implements the display function through the GPU, the display screen 740, and the application processor, etc.

[0124] In the sensor module 750, the pressure sensor 750A is used to sense pressure signals and can convert the pressure signals into electrical signals. The touch sensor 750B, also known as a "touch control device". The touch sensor 750B can be disposed on the display screen 740, and the touch sensor 750B and the display screen 740 together form a touch screen, also known as a "touch control screen". The touch sensor 750B is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event.

[0125] Another embodiment of the present application further provides a computer-readable storage medium storing instructions that, when run on a computer or a processor, cause the computer or the processor to execute one or more steps in any of the above methods.

[0126] The computer-readable storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0127] Another embodiment of the present application further provides a computer program product containing instructions. When the computer program product runs on a computer or a processor, it causes the computer or the processor to execute one or more steps in any of the above methods.

Claims

1. A method for capturing an image in a video, characterized in that, Including: Display a video recording interface of a camera application, where the dynamic range of the image displayed in the video recording interface is greater than a first value, the first value indicating that the image is not a low-dynamic-range image, and the image is stored in a buffer; In response to a capture instruction during video recording, use the image in the buffer corresponding to the capture moment as the captured image, where the capture moment is the moment corresponding to the capture instruction.

2. The method for capturing an image from a video according to claim 1, wherein The images stored in the buffer are obtained by performing high-dynamic-range (HDR) fusion and compression processing on a first Raw image and a second Raw image collected by an image sensor, and the exposure duration of the first Raw image is greater than that of the second Raw image.

3. The method for capturing an image from a video according to claim 2, wherein Before using the image in the buffer corresponding to the capture moment as the captured image, it further includes: Process the image in the buffer corresponding to the capture moment, and use the processed image in the buffer corresponding to the capture moment as the captured image, where the dynamic range of the captured image is greater than a second value, and the second value is greater than the first value.

4. The method for capturing an image from a video according to claim 3, wherein The processing of the image in the buffer corresponding to the capture moment includes: Perform format conversion, decompression processing, and image processing on the image in the buffer corresponding to the capture moment.

5. The method for capturing an image from a video according to any one of claims 1 to 4, characterized in that, The video recording interface displays a first button. The step of, in response to a capture instruction during video recording, using the image in the buffer corresponding to the capture moment as the captured image, where the capture moment is the moment corresponding to the capture instruction, includes: In response to a user's click operation on the first button, use the image in the buffer corresponding to the click moment of the first button as the captured image.

6. The method for capturing an image in a video according to any one of claims 1 to 4, characterized in that, The step of, in response to a capture instruction during video recording, using the image in the buffer corresponding to the capture moment as the captured image, where the capture moment is the moment corresponding to the capture instruction, includes: In response to an instruction for identifying a highlight moment, use the image in the buffer corresponding to the highlight moment as the captured image.

7. The method for capturing an image from a video according to any one of claims 1 to 6, characterized in that, The method is applied to an electronic device, which includes a first image sensor and a second image sensor, and the images stored in the buffer are sourced from the first image sensor.

8. The method for capturing an image from a video according to claim 7, characterized in that, After using the image in the buffer corresponding to the capture moment as the captured image in response to a capture instruction during video recording, it further includes: Receive an operation to switch cameras; In response to the operation to switch cameras, control the first image sensor to stop operating, and control the second image sensor to operate, where the image of the second image sensor forms the video recording interface.

9. The method for capturing an image from a video according to claim 8, wherein The controlling the second image sensor to operate includes: controlling the second image sensor to operate in a non-HDR mode, where the dynamic range of the image collected by the second image sensor in the non-HDR mode is less than the first value.

10. The method for capturing an image from a video according to claim 9, characterized in that, It further includes: When it is detected that the electronic device is in a high-brightness environment, control the second image sensor to operate in an HDR mode, where the dynamic range of the image collected by the second image sensor in the HDR mode is greater than the first value.

11. An electronic device, characterized in that, Including: One or more processors, a memory, a display screen, and an image sensor; The memory, the image sensor, and the display screen are coupled to the one or more processors. The memory is used for storing a computer program, and the computer program includes computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method for capturing an image during video recording according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, For storing a computer program, when the computer program is executed, it is specifically used to implement the method for capturing an image during video recording according to any one of claims 1 to 10.

Images