Video processing method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a video processing method and device, electronic equipment and a computer readable storage medium, which can improve the video definition and the dynamic range. In the method, the electronic equipment firstly controls an image sensor to alternately perform exposure of each frame according to a first frame exposure parameter and a second frame exposure parameter to obtain an image pair of each frame output by the image sensor; the first frame exposure parameter is used for indicating the image sensor to output an image pair comprising a first type of exposure image and a second type of exposure image, and the second frame exposure parameter is used for indicating the image sensor to output an image pair comprising the second type of exposure image and a third type of exposure image, or a fourth type of exposure image and a fifth type of exposure image; and multi-frame image fusion processing is carried out according to the at least three types of exposure images with different exposure degrees to obtain an image frame, and the at least three types of exposure images with different exposure degrees comprise at least one exposure image in the image pair of the current frame and the image pair of the adjacent frame.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to a video processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the increasing popularity of electronic devices such as mobile phones and increasingly powerful shooting functions, using mobile phones and other electronic devices to shoot videos has gradually become a trend.

[0003] When mobile phones and other electronic devices use image sensors to capture video, the dynamic range (DR) of the image sensor's perceived brightness is smaller than the real-world dynamic range, making it difficult to capture rich details and gradations in a single exposed image. Mobile phones and other electronic devices can use High Dynamic Range (HDR) technology to achieve improved detail and gradations by fusing multiple images with varying exposure levels.

[0004] However, when the image sensor capabilities of electronic devices such as mobile phones are low, the image sensor can only output two images with different exposure levels in one frame, which results in the clarity and dynamic range of the output frames obtained by multi-frame fusion still being low, resulting in low clarity and dynamic range of the captured video. Summary of the Invention

[0005] Embodiments of the present application provide a video processing method, apparatus, electronic device, and computer-readable storage medium, which can solve the problem of low video clarity and dynamic range caused by low image sensor capabilities of the electronic device.

[0006] In a first aspect, embodiments of the present application provide a video processing method for an electronic device comprising an image sensor. In this method, after the electronic device enters a video recording mode or a video recording process in response to a first operation, the electronic device first controls the image sensor to alternately expose each frame according to a first frame exposure parameter and a second frame exposure parameter, obtains image pairs of each frame output by the image sensor, and then performs multi-frame image fusion processing based on a target exposure image set to obtain an output frame. The first frame exposure parameter is used to instruct the image sensor to output an image pair comprising a first-type exposure image and a second-type exposure image, and the second frame exposure parameter is used to instruct the image sensor to output an image pair comprising a second-type exposure image and a third-type exposure image, or a fourth-type exposure image and a fifth-type exposure image; the first-type exposure image, the second-type exposure image, and the third-type exposure image have different exposure levels, or the first-type exposure image, the second-type exposure image, the fourth-type exposure image, and the fifth-type exposure image have different exposure levels; the image pair comprises two exposure images having different exposure levels; and the target exposure image set comprises at least three exposure images having different exposure levels, wherein the at least three exposure images having different exposure levels include the image pair of the current frame and at least one exposure image in the image pair of an adjacent frame.

[0007] As can be seen from the above, the embodiment of the present application first controls the image sensor to alternately expose each frame according to the first frame exposure parameters and the second frame exposure parameters during the exposure process, so that even if the image sensor only outputs two exposure images with different exposure levels in each frame, due to the different exposure strategies of adjacent frames, at least three types of exposure images with different exposure levels can be obtained; then, from the exposure images of two adjacent frames, at least three types of exposure images with different exposure levels are obtained for multi-frame fusion processing, thereby improving the dynamic range and clarity of the output frames, and thus making the dynamic range and clarity of the video obtained based on the output frames higher.

[0008] That is to say, even if the image sensor has low capabilities and can only output two exposure images with different exposure levels in one frame, the embodiment of the present application can obtain more exposure images with different exposure levels by coordinating exposure control and algorithm so that the exposure strategies of two adjacent frames are different; and at least three types of exposure images with different exposure levels of two adjacent frames are selected for multi-frame fusion, so that the exposure level categories and quantity of the fused exposure images are more, thereby making the dynamic range and clarity of the obtained image frames higher.

[0009] In one possible implementation of the first aspect, in the process of controlling an image sensor to alternately expose each frame according to first and second frame exposure parameters to obtain image pairs output by the image sensor for each frame, the electronic device controls the image sensor to alternately expose the current frame according to an exposure method of alternating row-by-row exposure, based on the first frame exposure parameters, to obtain an image pair output by the image sensor comprising a first-type exposure image and a second-type exposure image; and controls the image sensor to alternately expose the frame following the current frame according to an exposure method of alternating row-by-row exposure, based on the second frame exposure parameters, to obtain an image pair output by the image sensor comprising a second-type exposure image and a third-type exposure image, or a fourth-type exposure image and a fifth-type exposure image. In this way, by controlling two adjacent frames (i.e., the current frame and the adjacent frame) to be exposed according to different frame exposure parameters, i.e., by exposing them according to different exposure strategies, a larger number of exposure images with different exposure levels can be obtained even when the image sensor has low capabilities. This allows for subsequent use of the larger number of exposure images with different exposure levels for multi-frame fusion, thereby improving the dynamic range and clarity of the video.

[0010] In a possible implementation of the first aspect, the first type of exposure image is a long-exposure image, the second type of exposure image is a normal-exposure image, and the third type of exposure image is a short-exposure image; or, the first type of exposure image is a normal-exposure image, the second type of exposure image is a short-exposure image, and the third type of exposure image is a long-exposure image; or, the first type of exposure image is a short-exposure image, the second type of exposure image is a long-exposure image, and the third type of exposure image is a normal-exposure image; wherein the exposure time of the long-exposure image is greater than that of the normal-exposure image, and the exposure time of the normal-exposure image is greater than that of the short-exposure image. In this way, when the control frames are alternately exposed according to different exposure strategies, two adjacent frames have one exposure image with the same exposure level and three exposure images with different exposure levels (i.e., long exposure, normal exposure, and short exposure). This not only obtains more types of exposure images with different exposure levels to improve the dynamic range and clarity of the video, but also balances system power consumption, keeping it within a reasonable range.

[0011] In one possible implementation of the first aspect, when performing multi-frame image fusion processing based on a target exposure image set to obtain output frames, the electronic device may input each exposure image in the target exposure image set and target information corresponding to each exposure image into a multi-frame fusion model to obtain output frames output by the multi-frame fusion model. In this way, the multi-frame fusion model can effectively fuse the multiple exposure image frames.

[0012] In a possible implementation of the first aspect, when the second frame exposure parameter is used to instruct the image sensor to output a second-category exposure image and a third-category exposure image, the target exposure image set includes the first-category exposure image and the second-category exposure image of the current frame, and the third-category exposure image of an adjacent frame; or the first-category exposure image and the second-category exposure image of the current frame, and the second-category exposure image and the third-category exposure image of an adjacent frame;

[0013] When the second frame exposure parameter is used to instruct the image sensor to output the fourth and fifth exposure images, the target exposure image set includes the first and second exposure images of the current frame, and the fourth and fifth exposure images of the adjacent frame.

[0014] In a possible implementation manner of the first aspect, each exposure image in the target exposure image set is a RAW image, and the output frame is a RAW image or an RGB image.

[0015] In a possible implementation of the first aspect, after recording a video or entering a video recording mode, the electronic device can perform multi-frame fusion based on the exposure image obtained by exposure to obtain multiple output frames; when exiting the video recording in response to a second operation, a high dynamic range video file can be obtained, which is generated based on each output frame.

[0016] In a second aspect, an embodiment of the present application provides a video processing device, including a video recording module, an exposure control module, an image sensor and a multi-frame fusion module.

[0017] The video recording module is used to respond to the first operation and enter the video recording mode or video recording process.

[0018] The exposure control module is used to control the image sensor to alternately expose each frame according to a first frame exposure parameter and a second frame exposure parameter, and obtain image pairs of each frame output by the image sensor. The first frame exposure parameter is used to instruct the image sensor to output an image pair including a first-type exposure image and a second-type exposure image, and the second frame exposure parameter is used to instruct the image sensor to output an image pair including a second-type exposure image and a third-type exposure image, or a fourth-type exposure image and a fifth-type exposure image. The first-type exposure image, the second-type exposure image, and the third-type exposure image have different exposure levels, or the first-type exposure image, the second-type exposure image, the fourth-type exposure image, and the fifth-type exposure image have different exposure levels, and the image pair includes two exposure images with different exposure levels.

[0019] The multi-frame fusion module is used to perform multi-frame image fusion processing based on the target exposure image set to obtain an output frame. The target exposure image set includes at least three exposure images with different exposure levels. The at least three exposure images with different exposure levels include an image pair of the current frame and at least one exposure image in an image pair of an adjacent frame.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any one of the first aspects above when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the above-mentioned first aspects.

[0022] In a fifth aspect, embodiments of the present application provide a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement any of the methods described in the first aspect. The chip system can be a single chip or a chip module consisting of multiple chips.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute any of the methods described in the first aspect above.

[0024] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the video processing process provided in an embodiment of the present application;

[0026] Figure 2A A schematic diagram of a DOL two-exposure strategy provided in an embodiment of the present application;

[0027] Figure 2B A schematic structural diagram of the image sensor 110 provided in an embodiment of the present application;

[0028] Figure 2C A schematic diagram of row-by-row alternating exposure of a pixel array provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of the three-exposure strategy processing provided in an embodiment of the present application;

[0030] Figure 4A A schematic diagram of an exposure strategy and image output strategy for the video processing solution provided in an embodiment of the present application;

[0031] Figure 4B A schematic diagram of the exposure and readout process of the first frame provided in an embodiment of the present application;

[0032] Figure 4C A schematic diagram of the exposure and readout process of the second frame provided in an embodiment of the present application;

[0033] Figure 5A A schematic diagram of another exposure strategy and image output strategy for the video processing solution provided in an embodiment of the present application;

[0034] Figure 5B A schematic diagram of another exposure strategy and image output strategy for the video processing solution provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of another exposure strategy and image output strategy for the video processing solution provided in an embodiment of the present application;

[0036] Figure 7A A schematic diagram of multi-frame fusion using a model provided in an embodiment of the present application;

[0037] Figure 7B Another schematic diagram of multi-frame fusion using a model provided in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present application;

[0039] Figure 9 A schematic flow chart of a video processing method provided in an embodiment of the present application;

[0040] Figure 10 A schematic diagram of a video recording scenario provided in an embodiment of the present application;

[0041] Figure 11 This is a schematic block diagram of a video processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the following description, for the purpose of illustration rather than limitation, specific details such as particular system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0043] For example, see Figure 1The schematic diagram of the video processing process provided by the embodiment of the present application is shown. During the HDR video recording process, the image signal processor (ISP) 120 controls the image sensor (sensor) 110 to perform multiple exposures through the automatic exposure (AE) algorithm to obtain multiple original images (RAW images) with different exposure levels; the multiple images with different exposure levels are subjected to multi-frame fusion processing to obtain an HDR image; and an HDR video can be generated based on the multiple frames of HDR images.

[0044] During the automatic exposure control process, the AE algorithm can determine the exposure parameters according to the brightness of the current environment, and control the image sensor to perform multiple exposures according to the exposure parameters to obtain multiple exposure images with different exposure levels.

[0045] The exposure parameters include aperture, exposure time, and ISO (International Organization for Standardization).

[0046] The aperture value (e.g. F16) controls the size of the physical aperture opening, and thus the amount of light entering the imaging system. In comparison, when more light enters, the image is brighter.

[0047] Sensitivity is a measure of a photosensitive element's sensitivity to light. A high sensitivity means the element receives more light, resulting in a brighter image. Conversely, a low sensitivity means the element receives less light, resulting in a lower brightness.

[0048] Exposure time refers to the time interval between the camera's opening and closing of the shutter. By controlling the shutter speed, you can control exposure time, and thus the amount of light entering. Faster shutter speeds shorten exposure time and reduce light intake; slower shutter speeds increase exposure time and increase light intake. It's easy to understand that controlling both aperture and exposure time can control the amount of light entering.

[0049] Shutters can include global shutters (GS) and rolling shutters (RS). When an image sensor uses a global shutter for exposure, the entire pixel array is exposed at the same time; when a rolling shutter is used for exposure, the entire pixel array is exposed at different times, for example, pixels are exposed row by row.

[0050] The exposure level of the exposure image can be represented by exposure values ​​(EV). The exposure value of the obtained exposure image can be controlled by controlling at least one parameter among aperture, exposure time and sensitivity.

[0051] In the related art, when ISP 120 controls image sensor 110 to perform multiple exposures to obtain multiple exposure images, the capture of the previous frame must be completed before the capture of the next frame can begin. That is, the exposure of the next frame can only begin after the exposure and readout of the previous frame are complete. The completion of the exposure and readout of the previous frame means that all pixels in the previous frame have been exposed and the image data of all pixels has been read out. In this case, the exposure of the next frame must wait until all pixels in the previous frame have been read out before it can begin, resulting in a longer time interval between frames.

[0052] When the time interval between frames is long, it is easy to cause "ghosting" problems in the HDR image of multiple frames. Specifically, during the shooting process, if the camera shakes or there are moving objects in the shooting scene, the long time interval between frames will cause the position of the same object to be different in multiple exposure images. In this case, if the multiple exposure images are aligned and then fused, the "ghosting" problem will occur in the fused HDR image.

[0053] To shorten the time interval between frames and reduce the possibility of "ghosting" problems, related technologies usually control the image sensor to output multiple exposure images with different exposure levels in each frame, and fuse the multiple exposure images of each frame into a single output frame. In this case, the image sensor does not need to wait until the previous frame is read out before starting the next frame. Instead, when each row of pixels is exposed and read out, the next exposure will start immediately, thereby reducing the time interval between frames.

[0054] In the embodiments of the present application, a frame may refer to the exposure process of one frame, which may also be referred to as a shooting process or an acquisition cycle. That is, during a shooting process, the image sensor performs multiple exposures and outputs multiple exposure images with different exposure levels.

[0055] For example, technologies such as Digital Overlap High Dynamic Range (DOLHDR) and Stagger High Dynamic Range (stagger HDR) can output multiple images with different exposure levels during a single capture process and fuse them together to create an HDR image. Technologies such as DOL and stagger can output multiple images with different exposure levels quasi-simultaneously within a single frame, shortening the time interval between frames.

[0056] The following combination Figures 2A to 2C , which exemplarily introduces the process of two-exposure DOL. Figure 2A This is a schematic diagram of the DOL two-exposure strategy provided in an embodiment of the present application. Figure 2B is a schematic structural diagram of the image sensor 110 provided in an embodiment of the present application, Figure 2C This is a schematic diagram of row-by-row alternating exposure of a pixel array provided in an embodiment of the present application.

[0057] like Figure 1 and Figure 2A As shown, ISP 120 controls image sensor 110 to output an exposure L frame and an exposure S frame in each frame according to the dual exposure strategy. The exposure L frame refers to a long-exposure image, and the exposure S frame refers to a short-exposure image. The exposure duration of the exposure L frame is longer than that of the exposure S frame.

[0058] exist Figure 2A In the image processing, the image sensor 110 sequentially exposes the first frame (Frame 1, F1), the second frame (Frame 2, F2), the third frame (Frame 3, F3), and the fourth frame (Frame 4, F4) in chronological order, and outputs an exposure L frame and an exposure S frame at F1, F2, F3, and F4. After the ISP 120 obtains the exposure L frame and exposure S frame of each frame, it performs multi-frame fusion to obtain an output frame. That is, the exposure L frame and exposure S frame of F1 are fused to obtain output frame 1; the exposure L frame and exposure S frame of F2 are fused to obtain output frame 2; the exposure L frame and exposure S frame of F3 are fused to obtain output frame 3; and the exposure L frame and exposure S frame of F4 are fused to obtain output frame 4. Output frames 1 to 4 are all HDR images.

[0059] In each frame, image sensor 110 uses a rolling shutter to perform alternating row-by-row exposure according to two different exposure parameters: a long-exposure image and a short-exposure image. It will be appreciated that a rolling shutter exposes row by row, so after the first row of pixels in the pixel array is exposed, the second row of pixels is exposed, and so on, until all pixels in the pixel array are exposed. For a given row of pixels in image sensor 110, it is first exposed according to one exposure parameter and then according to the other exposure parameter.

[0060] like Figure 2B and Figure 2C As shown, the image sensor 110 may include a pixel array 111, a row selection circuit 113, and a readout circuit 112. The pixel array 111 includes n rows of pixels, namely, the first to nth rows of pixels, where n is a positive integer greater than 1.

[0061] The image sensor 110 can select pixel rows through the row selection circuit 113 according to the automatic exposure control of the ISP 120 to perform row-by-row alternating exposure, and read out image data through the readout circuit 112 .

[0062] Specifically, for the first row of pixels in the pixel array 111, the row selection circuit 113 is first used to control the first row of pixels in the pixel array 111 to perform long exposure according to the exposure parameters of the long exposure image, and the exposure of the first row of pixels is read out through the readout circuit 112 to obtain the long exposure image data L1 of the first row of pixels; then, according to the exposure parameters of the short exposure image, the row selection circuit 113 is used to control the first row of pixels in the pixel array 111 to perform short exposure, and the exposure of the first row of pixels is read out through the readout circuit 112 to obtain the short exposure image data S1 of the first row of pixels.

[0063] For the second row of pixels in the pixel array 111, the row selection circuit 113 is first used to control the second row of pixels in the pixel array 111 to perform long exposure according to the exposure parameters of the long exposure image, and the exposure of the second row of pixels is read out through the readout circuit 112 to obtain the long exposure image data L2 of the second row of pixels; then, according to the exposure parameters of the short exposure image, the row selection circuit 113 is used to control the second row of pixels in the pixel array 111 to perform short exposure, and the exposure of the second row of pixels is read out through the readout circuit 112 to obtain the short exposure image data S2 of the second row of pixels.

[0064] It is understood that after the long exposure of the first row of pixels in the pixel array 111 is completed, not only will the short exposure be performed on the first row of pixels, but the long exposure will also be performed on the next row of pixels (i.e., the second row). In addition, because the long exposure duration is longer than the short exposure duration, the short-exposure image data S1 obtained by the short exposure of the first row of pixels will be output earlier than the long-exposure image data L2 obtained by the long exposure of the second row of pixels. In other words, from the perspective of image data readout, after L1 is acquired, S1 is acquired first, followed by L2.

[0065] In addition, since short exposure is performed immediately after the first row of pixels completes long exposure, the interval between the exposure L frame and the exposure S frame is small, and the long exposure image and the short exposure image can be output "quasi-simultaneously" in one frame.

[0066] Similarly, according to the exposure parameters of the long exposure image and the exposure parameters of the short exposure image, long exposure and short exposure are alternately performed on the pixels in the third row, and long exposure image data L3 and short exposure image data S3 of the pixels in the third row are obtained.

[0067] Similarly, the pixels in the 4th row to the nth row are alternately exposed row by row, and the long-exposure image data and the short-exposure image data of each row of pixels are obtained in sequence, namely L4, S4...Ln, Sn.

[0068] From the perspective of image data readout, the long exposure image data and short exposure image data of each pixel row are output sequentially in a line interleaving manner. Figure 2C As shown, the output order of image data is: L1, S1, L2, S2, L3, S3, L4, S4...Ln, Sn.

[0069] When the long exposure and short exposure of the nth row are completed, that is, the long exposure and short exposure of all pixels in the pixel array 111 are completed, it can be considered that the exposure of one frame is completed. Based on the long exposure image data and short exposure image data of all pixel rows in one frame, the exposure L frame and the exposure S frame are obtained, that is, the long exposure image and short exposure image of the frame are obtained.

[0070] like Figure 2C As shown, based on the long-exposure image data of the 1st to nth rows, an exposure L frame is obtained; based on the short-exposure image data of the 1st to nth rows, an exposure S frame is obtained.

[0071] Among them, the long exposure image data L1 of the first row is used as the first row of the exposure L frame; the long exposure image data L2 of the second row is used as the second row of the exposure L frame; the long exposure image data L3 of the third row is used as the third row of the exposure L frame; the long exposure image data L4 of the fourth row is used as the fourth row of the exposure L frame; and so on, the long exposure image data Ln of the nth row is used as the nth row of the exposure L frame.

[0072] Similarly, the short exposure image data S1 of the first row is used as the first row of the exposure S frame; the short exposure image data S2 of the second row is used as the second row of the exposure S frame; the short exposure image data S3 of the third row is used as the third row of the exposure S frame; the short exposure image data S4 of the fourth row is used as the fourth row of the exposure S frame; and so on, the short exposure image data Sn of the nth row is used as the nth row of the exposure S frame.

[0073] It is understandable that Figure 2A Each frame of F1, F2, F3 and F4 can be Figure 2C The exposure is performed alternately row by row in the manner shown, so as to obtain an exposure L frame and an exposure S frame with a short time interval in one frame.

[0074] It should be pointed out that, due to limitations of computing power, hardware, and other factors, most HDR video solutions control the image sensor to output two images with different exposure levels (e.g., a long-exposure image and a short-exposure image) in one frame, and fuse the two exposure images of each frame into an output frame to reduce the occurrence of "ghosting" problems. However, the dynamic range and image details of the two images with different exposure levels are limited, resulting in low clarity and dynamic range of the fused output frame, and poor video quality. In order to further improve the clarity and dynamic range of the video and obtain better video effects, a more capable image sensor can be used to perform three exposures in one shooting process (i.e., frame) to obtain three images with different exposure levels, and fuse the three images with different exposure levels of each frame into an output frame. This allows for the fusion of more images with different exposure levels and a greater number of images, resulting in higher video clarity and dynamic range.

[0075] More capable image sensors can support the quasi-simultaneous output of three images with different exposure levels in a single frame. For example, a long-exposure image, a medium-exposure image (or normal-exposure image), and a short-exposure image can be output in a single frame, with the medium-exposure image (or normal-exposure image) having an exposure time between the long-exposure and short-exposure images.

[0076] For example, see Figure 3 The following diagram illustrates a triple-exposure strategy processing scheme provided by an embodiment of the present application. The image sensor outputs three exposure images with different exposure levels: the first frame F1, the second frame F2, and the third frame F1. The S, N, and L frames of F1 are fused to produce output frame 1. The S, N, and L frames of F2 are fused to produce output frame 2. The N frame refers to a normal exposure image or a medium exposure image.

[0077] In each frame, the image sensor can perform alternating exposure line by line using a rolling shutter method according to three different exposure parameters. The three different exposure parameters include long exposure parameter, normal exposure parameter, and short exposure parameter. Figure 2C Similar to the row-by-row alternating exposure of the CMOS image sensor, a row of pixels in the pixel array is first exposed and read out using the long exposure parameters, then exposed and read out using the normal exposure parameters, and finally exposed and read out using the short exposure parameters. In addition, after completing the long exposure of a row of pixels, not only does the normal exposure and short exposure continue for the current row of pixels, but the long exposure also continues for the next row of pixels.

[0078] Although using a higher-capability image sensor to obtain three images with different exposure levels in each frame can improve video clarity and dynamic range, it is necessary to change the device hardware. That is, the original lower-capability image sensor needs to be replaced with a higher-capability image sensor, and other related hardware may also need to be adaptively changed to adapt to the higher-capability image sensor. For example, other related hardware includes a buffer. A higher-capability image sensor requires a higher-speed buffer to better cache the image data of each pixel row. Changes in device hardware will bring about related issues such as increased device hardware costs. In other words, the relevant technology is based on changes in device hardware to achieve improvements in video dynamic range and clarity.

[0079] During the research, the inventors found that in the related art, the exposure strategy of the image sensor in each frame is the same. If the exposure strategy of two adjacent frames is the same, the image sensor with lower capabilities will obtain fewer image categories with different exposure levels, which will result in lower video clarity and dynamic range. For example, Figure 2A The exposure strategies of F1 and F2 are the same, both of which perform long exposure and short exposure to obtain exposure L frames and exposure S frames, that is, only two types of exposure images with different exposure degrees, long exposure and short exposure, can be obtained.

[0080] Based on this, an embodiment of the present application provides a video processing solution. By controlling the image sensor to use different exposure strategies between two adjacent frames, even with a lower image sensor capability, a wider range of exposure images with varying degrees of exposure can be acquired. Furthermore, by selecting at least three exposure images with varying degrees of exposure from two adjacent frames for fusion, the fused images have a greater number of exposure categories and levels, thereby improving video clarity and dynamic range. This solution improves video clarity and dynamic range by leveraging the capabilities of existing device hardware without changing the device hardware. This solution leverages the synergy between exposure and fusion algorithms to achieve the desired effect.

[0081] That is to say, in the related art, when the image sensor capability is low, the image sensor outputs a long-exposure image and a short-exposure image in each frame, and the image sensor has the same exposure strategy in each frame, only two types of exposure images with different exposure levels can be obtained; in this way, in the subsequent multi-frame fusion process, only the long-exposure and short-exposure images of each frame with different exposure levels are fused, resulting in poor dynamic range and clarity of the fused output frames.

[0082] In the video processing solution of the embodiment of the present application, the exposure strategies of two adjacent frames are different by controlling the image sensor to alternately expose each frame according to the exposure parameters of the first frame and the exposure parameters of the second frame. In this way, even if the image sensor has a low capability, more exposure images with different exposure levels (at least three categories) can be obtained based on the capabilities of existing device hardware; in the subsequent multi-frame fusion process, at least three categories of images with different exposure levels of two adjacent frames are selected for fusion. In this way, a larger number of exposure images are fused, making the output frame clearer; and a larger number of exposure images with different exposure levels (at least three categories) are fused, making the dynamic range of the output frame higher, thereby improving the clarity and dynamic range of the video.

[0083] The first frame exposure parameter and the second frame exposure parameter both refer to frame exposure parameters. A frame exposure parameter may indicate the number of exposure images output by the image sensor for that frame and the exposure level of each exposure image. For example, a frame exposure parameter may instruct the image sensor to output two exposure images: a long exposure image and a short exposure image.

[0084] The first frame exposure parameters and the second frame exposure parameters are different. Typically, the image sensor outputs the same number of exposure images in each frame. For example, the image sensor outputs two exposure images in each frame. Therefore, the first frame exposure parameters indicating the number of exposure images output by the image sensor in the frame are the same as the second frame exposure parameters indicating the number of exposure images output by the image sensor in the frame. Based on this, the difference between the first frame exposure parameters and the second frame exposure parameters can mean that the two frame exposure parameters indicate different exposure levels for the exposure images output by the image sensor.

[0085] Furthermore, the exposure level of the exposure image output by the first frame exposure parameter indication image sensor is not completely the same as the exposure level of the exposure image output by the second frame exposure parameter indication image sensor. The exposure levels of the exposure images are not completely the same may mean that the exposure levels of all exposure images output by the image sensor indicated by the two frame exposure parameters are different, or the exposure levels of some exposure images are the same.

[0086] For example, the first frame exposure parameters instruct the image sensor to output a long-exposure image and a medium-exposure image in the current frame, and the second frame exposure parameters instruct the image sensor to output a normal-exposure image and a short-exposure image in the next frame. In this case, the exposure duration of the long-exposure image is longer than that of the medium-exposure image, the exposure duration of the medium-exposure image is longer than that of the normal-exposure image, and the exposure duration of the normal-exposure image is longer than that of the short-exposure image. The long-exposure image, the medium-exposure image, the normal-exposure image, and the short-exposure image have different exposure levels. That is, the first frame exposure parameters and the second frame exposure parameters indicate that all exposure images output by the image sensor have different exposure levels.

[0087] For another example, the first frame exposure parameter instructs the image sensor to output a long-exposure image and a normal-exposure image in the current frame, while the second frame exposure parameter instructs the image sensor to output a short-exposure image and a normal-exposure image in the next frame. In this case, the two frame exposure parameters instruct the image sensor to output the partially exposed images with the same exposure level, meaning that both the current and next frames output normal-exposure images.

[0088] Alternating the exposure of each frame according to the first frame exposure parameters and the second frame exposure parameters may mean: exposing the first frame using the first frame exposure parameters, exposing the second frame using the second frame exposure parameters, exposing the third frame using the first frame exposure parameters, and exposing the fourth frame using the second frame exposure parameters; and exposing each frame alternately in this manner. Of course, it is also possible to first expose the second frame exposure parameters and then expose the first frame exposure parameters.

[0089] It can be understood that since the exposure parameters of the first frame and the exposure parameters of the second frame indicate that the exposure levels of the exposure images output by the image sensor are not exactly the same, by alternating the exposure according to the exposure parameters of the first frame and the exposure parameters of the second frame, the exposure levels of the exposure images of any two adjacent frames can be made not exactly the same, thereby obtaining more categories of exposure images with different exposure levels.

[0090] In some embodiments, both the first frame exposure parameter and the second frame exposure parameter indicate that the image sensor outputs two exposure images with different exposure levels in each frame, and the exposure level of one of the two exposure images output by the first frame exposure parameter is the same as the exposure level of one of the two exposure images output by the second frame exposure parameter.

[0091] In one optional embodiment, the exposure parameters of the first frame indicate that the image sensor outputs two exposure images as a long-exposure image and a normal-exposure image; while the exposure parameters of the second frame indicate that the image sensor outputs two exposure images as a normal-exposure image and a short-exposure image. In this case, among the four exposure images of two adjacent frames, there are three exposure images with different exposure levels: the long-exposure image, the normal-exposure image, and the short-exposure image. The normal-exposure image output by the exposure parameters of the first frame and the normal-exposure image output by the exposure parameters of the second frame have the same exposure level.

[0092] For example, see Figure 4A A schematic diagram of an exposure strategy and an image output strategy of a video processing solution provided in an embodiment of the present application is shown. In chronological order, the image sensor alternately exposes the first frame (i.e., F1), the second frame (i.e., F2), the third frame (i.e., F3), and the third frame (i.e., F4) according to the first frame exposure parameters and the second frame exposure parameters.

[0093] Specifically, the image sensor exposes the first frame according to the first frame exposure parameters, and outputs exposure L frame and exposure N frame; exposes the second frame according to the second frame exposure parameters, and outputs exposure S frame and exposure N frame; exposes the third frame according to the first frame exposure parameters, and outputs exposure L frame and exposure N frame; exposes the fourth frame according to the second frame exposure parameters, and outputs exposure S frame and exposure N frame.

[0094] It is understood that the image sensor may also first expose according to the second frame exposure parameters and then expose according to the first frame exposure parameters. In this case, the first and third frames are exposed to obtain exposure frames S and N, and the second and third frames are exposed to obtain exposure frames L and N.

[0095] At this time, the first frame exposure parameters include a first exposure parameter and a second exposure parameter, the first exposure parameter is used to expose to obtain exposure frame L, and the second exposure parameter is used to expose to obtain exposure frame N. The second frame exposure parameters include a second exposure parameter and a third exposure parameter, and the third exposure parameter is used to expose to obtain exposure frame S.

[0096] During the exposure process of the first frame, the image sensor is exposed according to the first exposure parameter and the second exposure parameter. Figure 4BThe schematic diagram of the exposure and readout process for the first frame provided by an embodiment of the present application shows a pixel array comprising rows 1 to n, and the pixel array of the image sensor uses a rolling shutter for alternating exposure row by row. For the pixels in the first row of the pixel array, the row selection circuit first controls the long exposure of the pixels in the first row of the pixel array according to a first exposure parameter, and the readout circuit reads out the exposure of the pixels in the first row of the pixel array, obtaining long-exposure image data L1 for the pixels in the first row of the pixel array. Then, the row selection circuit controls the normal exposure of the pixels in the first row of the pixel array according to a second exposure parameter, and the readout circuit reads out the exposure of the pixels in the first row of the pixel array, obtaining normal-exposure image data N1 for the pixels in the first row of the pixel array.

[0097] For the second row of pixels in the pixel array, first, according to the first exposure parameter, the row selection circuit controls the second row of pixels in the pixel array to perform long exposure, and the exposure of the second row of pixels is read out by the readout circuit to obtain long exposure image data L2 of the second row of pixels; then, according to the second exposure parameter, the row selection circuit controls the second row of pixels in the pixel array to perform normal exposure, and the exposure of the second row of pixels is read out by the readout circuit to obtain normal exposure image data N2 of the second row of pixels.

[0098] It is understandable that after the long exposure of the first row of pixels in the pixel array is completed, not only will normal exposure continue for the first row of pixels, but long exposure will also continue for the next row of pixels (i.e., the second row). In addition, because the long exposure duration is longer than the normal exposure duration, the normal exposure image data N1 obtained by the normal exposure of the first row of pixels will be output earlier than the long exposure image data L2 obtained by the long exposure of the second row of pixels. In other words, from the perspective of image data readout, after L1 is obtained, N1 will be obtained first, followed by L2.

[0099] Similarly, the pixels in the 3rd row to the nth row are alternately exposed row by row to obtain long exposure image data and normal exposure image data of each row of pixels, i.e., L3, N3, L4, N4...Ln, Nn.

[0100] From the perspective of image data readout, the long-exposure image data and the normal-exposure image data of each pixel row are output sequentially in a row-interleaved manner, that is, the output order of the image data is: L1, N1, L2, N2, L3, N3, L4, N4...Ln, Nn.

[0101] When the long exposure and normal exposure of the nth row are completed, that is, the long exposure and normal exposure of all pixels in the pixel array are completed, it can be considered that the exposure of the first frame is completed. Based on the long exposure image data and normal exposure image data of all pixel rows in the first frame, the exposure L frame and the exposure N frame are obtained.

[0102] like Figure 4BAs shown, based on the long exposure image data of the 1st to nth rows, the exposure L frame is obtained; based on the normal exposure image data of the 1st to nth rows, the exposure N frame is obtained.

[0103] Among them, the long exposure image data L1 of the first row is used as the first row of the exposure L frame; the long exposure image data L2 of the second row is used as the second row of the exposure L frame; the long exposure image data L3 of the third row is used as the third row of the exposure L frame; the long exposure image data L4 of the fourth row is used as the fourth row of the exposure L frame; and so on, the long exposure image data Ln of the nth row is used as the nth row of the exposure L frame.

[0104] The normal exposure image data N1 of the first row is used as the first row of the exposure N frames; the normal exposure image data N2 of the second row is used as the second row of the exposure N frames; the normal exposure image data N3 of the third row is used as the third row of the exposure N frames; the normal exposure image data N4 of the fourth row is used as the fourth row of the exposure N frames; and so on, the normal exposure image data Nn of the nth row is used as the nth row of the exposure N frames.

[0105] During the exposure process of the second frame, the image sensor is exposed according to the second exposure parameter and the third exposure parameter. Figure 4C The schematic diagram of the exposure and readout process of the second frame provided by the embodiment of the present application is shown. Figure 4A Similarly, the pixel array of the image sensor uses a rolling shutter to perform alternating exposure row by row to obtain exposure image data for each row of pixels. Figure 4C As shown in FIG, the exposure image data of each row of pixels is read in the order: N1, S1, N2, S2, N3, S3, N4, S4 ... Nn, Sn. Nn refers to the normal exposure image data of the n-th row of pixels, and Sn refers to the short exposure image data of the n-th row of pixels.

[0106] According to the normal exposure image data of the 1st to nth rows (ie, N1 to Nn), N exposure frames are obtained; according to the short exposure image data of the 1st to nth rows (ie, S1 to Sn), S exposure frames are obtained.

[0107] During the exposure process of the third frame, the image sensor performs exposure according to the first exposure parameters and the second exposure parameters. The specific process can be found in the exposure and readout process of the first frame above and will not be repeated here. During the exposure process of the fourth frame, the image sensor performs exposure according to the second exposure parameters and the third exposure parameters. The specific process can be found in the exposure and readout process of the second frame above and will not be repeated here.

[0108] By comparison Figure 4A and Figure 2A Available, Figure 2AThe exposure strategies of two adjacent frames are the same, and only two exposure images with different exposure levels can be obtained. For example, F1 to F4 are all exposed to obtain exposure L frame and exposure S frame.

[0109] and Figure 4A The exposure strategies of two adjacent frames in the image are different. For example, the exposure strategies of F1 and F2 are different. That is, exposure F1 results in exposure L and exposure N frames, while exposure F2 results in exposure S and exposure N frames. The exposure strategies of F2 and F3 are also different. That is, exposure F2 results in exposure S and exposure N frames, while exposure F3 results in exposure L and exposure N frames. In this way, even if the image sensor's capability only supports outputting two images with different exposure levels in a single frame, due to the different exposure strategies of two adjacent frames, three types of exposure images with different exposure levels can be obtained: exposure L, exposure S, and exposure N frames. This means that more exposure images with different exposure levels can be obtained.

[0110] In another optional embodiment, the exposure parameters of the first frame indicate that the image sensor outputs two exposure images as a long exposure image and a short exposure image; while the exposure parameters of the second frame indicate that the image sensor outputs two exposure images as a normal exposure image and a short exposure image. In this case, the four exposure images of two adjacent frames include three types of exposure images with different exposure levels: a long exposure image, a normal exposure image, and a short exposure image. The short exposure image output by the exposure parameters of the first frame and the short exposure image output by the exposure parameters of the second frame have the same exposure level.

[0111] For example, see Figure 5A A schematic diagram of another exposure strategy and image output strategy of the video processing solution provided in an embodiment of the present application is shown. In chronological order, the image sensor alternately exposes the first frame (i.e., F1), the second frame (i.e., F2), the third frame (i.e., F3), and the fourth frame (i.e., F4) according to the first frame exposure parameters and the second frame exposure parameters.

[0112] Specifically, the image sensor exposes the first frame according to the first frame exposure parameters, and outputs exposure L frame and exposure S frame; exposes the second frame according to the second frame exposure parameters, and outputs exposure S frame and exposure N frame; exposes the third frame according to the first frame exposure parameters, and outputs exposure L frame and exposure S frame; exposes the third frame according to the second frame exposure parameters, and outputs exposure S frame and exposure N frame.

[0113] It is understandable that the image sensor may also first expose according to the exposure parameters of the second frame, and then expose according to the exposure parameters of the first frame. In this case, the first and third frames are exposed to obtain exposure S frames and exposure N frames, and the second and fourth frames are exposed to obtain exposure L frames and exposure S frames.

[0114] By comparison Figure 5A and Figure 2A Available, Figure 5A The exposure strategies for two adjacent frames are different. Therefore, even if the image sensor only supports outputting two images with different exposure levels within a single frame, the different exposure strategies for the two adjacent frames can still produce three types of exposure images with different exposure levels: L-frame, S-frame, and N-frame. This allows for a wider range of exposure images. The exposure and readout process for each frame can be found above and will not be further elaborated here.

[0115] In another alternative embodiment, the first frame exposure parameters indicate that the image sensor outputs two exposure images as a long-exposure image and a short-exposure image; while the second frame exposure parameters indicate that the image sensor outputs two exposure images as a normal-exposure image and a long-exposure image. In this case, among the four exposure images of two adjacent frames, there are three exposure images with different exposure levels: the long-exposure image, the normal-exposure image, and the short-exposure image. The long-exposure image output by the first frame exposure parameters and the long-exposure image output by the second frame exposure parameters have the same exposure level.

[0116] For example, see Figure 5B A schematic diagram of another exposure strategy and image output strategy of the video processing solution provided by an embodiment of the present application is shown. In chronological order, the image sensor alternately exposes the first frame (i.e., F1), the second frame (i.e., F2), the third frame (i.e., F3), and the fourth frame (i.e., F4) according to the first frame exposure parameters and the second frame exposure parameters.

[0117] Specifically, the image sensor exposes the first frame according to the first frame exposure parameters, and outputs exposure L frame and exposure S frame; exposes the second frame according to the second frame exposure parameters, and outputs exposure L frame and exposure N frame; exposes the third frame according to the first frame exposure parameters, and outputs exposure L frame and exposure S frame; exposes the fourth frame according to the second frame exposure parameters, and outputs exposure L frame and exposure N frame.

[0118] It is understandable that the image sensor may also first expose according to the second frame exposure parameters and then expose according to the first frame exposure parameters. In this case, the first and third frames are exposed to obtain exposure frames N and L, and the second and fourth frames are exposed to obtain exposure frames L and S.

[0119] By comparison Figure 5B and Figure 2A Available, Figure 5B The exposure strategies of two adjacent frames are different. Even if the image sensor only supports outputting two images with different exposure levels in one frame, due to the different exposure strategies of the two adjacent frames, three types of exposure images with different exposure levels, namely, exposure L frame, exposure S frame, and exposure N frame, can be obtained, that is, more types of exposure images with different exposure levels can be obtained.

[0120] In the embodiment described above, the image sensor can output three types of exposure images with different exposure levels in two adjacent frames. In other embodiments, the image sensor can output four types of exposure images with different exposure levels in two adjacent frames. In this case, both the first frame exposure parameters and the second frame exposure parameters instruct the image sensor to output two exposure images with different exposure levels in each frame, and the exposure levels of the two exposure images output by the first frame exposure parameters are different from the exposure levels of the two exposure images output by the second frame exposure parameters.

[0121] In one optional embodiment, the first frame exposure parameters indicate that the image sensor outputs two exposure images: a long-exposure image and a normal-exposure image; the second frame exposure parameters indicate that the image sensor outputs two exposure images: a second-short-exposure image and a short-exposure image. In this case, the four exposure images of two adjacent frames include four exposure images with different exposure levels: a long-exposure image, a normal-exposure image, a second-short-exposure image, and a short-exposure image. The exposure time of the long-exposure image is longer than that of the normal-exposure image, the exposure time of the normal-exposure image is longer than that of the second-short-exposure image, and the exposure time of the second-short-exposure image is longer than that of the short-exposure image.

[0122] For example, see Figure 6 A schematic diagram of another exposure strategy and image output strategy of the video processing solution provided in an embodiment of the present application is shown. In chronological order, the image sensor alternately exposes the first frame (i.e., F1), the second frame (i.e., F2), the third frame (i.e., F3), and the fourth frame (i.e., F4) according to the first frame exposure parameters and the second frame exposure parameters.

[0123] Specifically, the image sensor exposes frame 1 according to the first frame exposure parameters, outputting exposure frames L and N. It exposes the second frame according to the second frame exposure parameters, outputting exposure frames S and D. It exposes the third frame according to the first frame exposure parameters, outputting exposure frames L and N. It exposes the fourth frame according to the second frame exposure parameters, outputting exposure frames S and D. Exposure frame D is the second shortest exposure image.

[0124] It is understood that the image sensor may also first expose according to the exposure parameters of the second frame, and then expose according to the exposure parameters of the first frame. In this case, the first and third frames are exposed to obtain the exposure S frame and the exposure D frame, and the second and fourth frames are exposed to obtain the exposure L frame and the exposure N frame.

[0125] By comparison Figure 6 and Figure 2A Available, Figure 6The exposure strategies of two adjacent frames are different. Even if the image sensor only supports outputting two images with different exposure levels in one frame, due to the different exposure strategies of the two adjacent frames, four types of exposure images with different exposure levels, namely, exposure L frame, exposure D frame, exposure S frame, and exposure N frame, can be obtained, that is, more exposure images with different exposure levels can be obtained.

[0126] After controlling the image sensor to expose according to different exposure strategies in two adjacent frames to obtain more categories of exposure images with different exposure levels, multiple frames can be fused based on the multiple exposure images with different exposure levels to obtain output frames.

[0127] In the embodiment of the present application, at least three types of exposure images with different exposure levels can be selected from the exposure images of two adjacent frames, and the at least three types of exposure images with different exposure levels can be fused to obtain an output frame.

[0128] In some embodiments, three exposure images are selected from exposure images of two adjacent frames for multi-frame fusion, and the exposure levels of the three exposure images are different.

[0129] For example, Figure 4A As shown, the exposure images output by the image sensor in two adjacent frames include exposure L frame, exposure N frame, exposure S frame and exposure N frame, a total of four exposure images. At this time, three images with different exposure levels, exposure L frame, exposure N frame and exposure S frame, are selected from the exposure images of two adjacent frames for multi-frame fusion. Specifically, the exposure L frame and exposure N frame of F1, and the exposure S frame of F2 are selected for multi-frame fusion to obtain output frame 1; the exposure S frame and exposure N frame of F2, and the exposure L frame of F3 are selected for multi-frame fusion to obtain output frame 2; the exposure L frame and exposure N frame of F3, and the exposure S frame of F4 are selected for multi-frame fusion to obtain output frame 3; the exposure S frame and exposure N frame of F4, and the exposure L frame of the next frame (not shown in the figure) are selected for multi-frame fusion to obtain output frame 4.

[0130] For example, Figure 5AAs shown, the exposure images output by the image sensor in two adjacent frames include exposure L frame, exposure S frame, exposure N frame and exposure S frame, a total of four exposure images. At this time, three images with different exposure levels, exposure L frame, exposure N frame and exposure S frame, are selected from the exposure images of two adjacent frames for multi-frame fusion. Specifically, the exposure L frame and exposure S frame of F1, and the exposure N frame of F2 are selected for multi-frame fusion to obtain output frame 1; the exposure S frame and exposure N frame of F2, and the exposure L frame of F3 are selected for multi-frame fusion to obtain output frame 2; the exposure L frame and exposure S frame of F3, and the exposure N frame of F4 are selected for multi-frame fusion to obtain output frame 3; the exposure S frame and exposure N frame of F4, and the exposure L frame of the next frame (not shown in the figure) are selected for multi-frame fusion to obtain output frame 4.

[0131] For example, Figure 5B As shown, the exposure images output by the image sensor in two adjacent frames include the exposure S frame, the exposure L frame, the exposure N frame and the exposure L frame, a total of four exposure images. At this time, three images with different exposure levels, namely the exposure L frame, the exposure N frame and the exposure S frame, are selected from the exposure images of the two adjacent frames for multi-frame fusion. Specifically, the exposure L frame and the exposure S frame of F1, and the exposure N frame of F2 are selected for multi-frame fusion to obtain the output frame 1; the exposure L frame and the exposure N frame of F2, and the exposure S frame of F3 are selected for multi-frame fusion to obtain the output frame 2; the exposure L frame and the exposure S frame of F3, and the exposure N frame of F4 are selected for multi-frame fusion to obtain the output frame 3; the exposure L frame and the exposure N frame of F4, and the exposure L frame of the next frame (not shown in the figure) are selected for multi-frame fusion to obtain the output frame 4.

[0132] Compared to Figure 2A The exposure L frame and exposure S frame of each frame are fused to obtain the output frame. The embodiment of the present application selects three images with different exposure levels, namely the exposure L frame, exposure N frame and exposure S frame of two adjacent frames, for multi-frame fusion, fusing a larger number of exposure images and images of more types of exposure levels, thereby making the obtained image frame have higher clarity and dynamic range.

[0133] In other embodiments, four exposure images may be selected from exposure images of two adjacent frames for multi-frame fusion, wherein the four exposure images include three types of images with different exposure levels, or four types of images with different exposure levels.

[0134] For example, the image sensor is configured as follows: Figure 4A 、 Figure 5A or Figure 5B The exposure strategy shown is used for exposure, and four exposure images are output in two adjacent frames, and the four exposure images include images of three types with different exposure levels. At this time, the four exposure images of two adjacent frames are selected and fused into the output frame.

[0135] Compared with the method of fusing only three exposure images of exposure S frame, exposure L frame and exposure N frame, the method of fusing four exposure images of two adjacent frames to obtain output frames has higher clarity because more exposure images are fused.

[0136] As another example, the image sensor is configured as follows: Figure 6 The exposure strategy shown is used for exposure, and four exposure images are output to two frames, and the exposure levels of the four exposure images are different. At this time, the exposure L frame, exposure N frame, exposure S frame and exposure D frame of the two adjacent frames are fused into an output frame. Specifically, the exposure L frame and exposure N frame of F1, and the exposure S frame and exposure D frame of F2 are multi-frame fused to obtain output frame 1; the exposure S frame and exposure D frame of F2, and the exposure L frame and exposure N frame of F3 are multi-frame fused to obtain output frame 2; the exposure L frame and exposure N frame of F3, and the exposure S frame and exposure D frame of F4 are multi-frame fused to obtain output frame 3; the exposure S frame and exposure D frame of F4, and the exposure L frame and exposure N frame of the next frame (not shown in the figure) are multi-frame fused to obtain output frame 4.

[0137] Compared with fusing three exposure images with different exposure levels into an output frame, fusing four exposure images with different exposure levels into an output frame has a higher dynamic range and clarity due to the fusion of exposure images with more exposure levels and a larger number of exposure images.

[0138] Of course, in addition to fusing three or four exposure images, five or six exposure images can also be fused, or even more exposure images, which is not limited here. Figure 4A 、 Figure 5A or Figure 5B When performing exposure using the exposure strategy shown, the exposure images of F1, F2, and F3 can be fused into an output frame, that is, six exposure images are fused.

[0139] It is understandable that the more exposure images that are fused, the higher the dynamic range and clarity of the output frame. However, in practical applications, the number of exposure images to be fused should usually be reasonably determined taking into account the performance constraints of the system.

[0140] In some embodiments, an artificial intelligence (AI) model may be used for image fusion during the multi-frame fusion process. After the model training is complete, multiple exposure images are input into the model to obtain output frames.

[0141] For example, see Figure 7AThis diagram illustrates a multi-frame fusion model implemented in an embodiment of the present application. Three images with different exposure levels—exposure L, exposure S, and exposure N—are input to the AI ​​model, along with additional information about each exposure image. The AI ​​model then fuses the three exposure images into a single output frame. This additional information may include, but is not limited to, current exposure levels, noise model, and other related information.

[0142] The exposure S frame, exposure L frame, and exposure N frame can be RAW images, and the output frame is an RGB image or RAW image. The AI ​​model can use a commonly used network model, such as the U-net model.

[0143] For example, see Figure 7B Another schematic diagram of using a model for multi-frame fusion provided by an embodiment of the present application is shown. Four images with different exposure levels, Exposure 1, Exposure 2, Exposure 3, and Exposure 4, as well as additional information of each exposure image frame are used as input to the AI ​​model. The AI ​​model performs multi-frame fusion based on the input data and outputs an output frame, that is, the four exposure images are fused into one output frame. Exemplarily, Exposure 1, Exposure 2, Exposure 3, and Exposure 4 can be: Exposure L frame, Exposure N frame, Exposure S frame, and Exposure D frame. Exposure 1, Exposure 2, Exposure 3, and Exposure 4 can be RAW images, and the output frame is an RGB image or a RAW image.

[0144] Of course, in other embodiments, in addition to using the model to perform multi-frame fusion, other multi-frame fusion methods can also be used to perform multi-frame fusion, which is not limited here.

[0145] After the output frames are fused, a video can be generated based on each output frame. Since the output frames have a higher dynamic range and clarity, the video generated from the output frames also has a higher dynamic range and clarity.

[0146] The video processing solution provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, or vehicle-mounted monitoring devices. The specific type and structure of the electronic device are not limited here.

[0147] For example, Figure 8 8 shows a schematic structural diagram of an electronic device 800. The electronic device 800 may include, but is not limited to, a processor 810, a memory 820, a camera 830, and a display screen 840.

[0148] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 800. In other embodiments of the present application, the electronic device 800 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. For example, when the electronic device 800 is specifically a mobile phone, it also includes a universal serial bus (USB) interface, a charging management module, a power management module, a battery, an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone jack, a sensor, a button, a motor, an indicator, and a subscriber identification module (SIM) card interface, etc.

[0149] The processor 810 may include one or more processing units. For example, the processor 810 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, and a digital signal processor (DSP). The different processing units may be independent devices or integrated into one or more processors.

[0150] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0151] The processor 810 may include one or more interfaces, such as a mobile industry processor interface (MIPI) and a general-purpose input / output (GPIO) interface.

[0152] The MIPI interface can be used to connect the processor 810 to peripheral devices such as the display screen 840 and the camera 830. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 810 and the camera 830 communicate via the CSI interface to implement the camera function of the electronic device 800. The processor 810 and the display screen 840 communicate via the DSI interface to implement the display function of the electronic device 800.

[0153] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 810 with the camera 830 and the display screen 840, etc.

[0154] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 800. In other embodiments of the present application, the electronic device 800 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0155] Electronic device 800 implements display functionality through a GPU, display screen 840, and an application processor. The GPU is a microprocessor for image processing that connects display screen 840 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 810 may include one or more GPUs that execute program instructions to generate or modify display information.

[0156] Display screen 840 is used to display images, videos, and the like. Display screen 840 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 800 may include one or N display screens 840, where N is a positive integer greater than one.

[0157] The electronic device 800 can implement a shooting function through an ISP, a camera 830, a video codec, a GPU, a display screen 840, and an application processor.

[0158] The ISP processes data fed back by the camera 830. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within the camera 830.

[0159] The camera 830 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 800 may include 1 or N cameras 830, where N is a positive integer greater than 1.

[0160] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0161] The memory 820 can be used to store computer executable program code, and the executable program code includes instructions. The memory 820 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 800. In addition, the memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 810 executes various functional applications and data processing of the electronic device 800 by running instructions stored in the memory 820 and / or instructions stored in a memory provided in the processor.

[0162] Optionally, the electronic device 800 may further include an ambient light sensor. The ambient light sensor may be used to sense the brightness of the ambient light. The electronic device 800 may automatically adjust the white balance or perform automatic exposure control during shooting based on the sensed brightness of the ambient light.

[0163] The video processing solution provided in the embodiment of the present application is introduced below using electronic device 800 as an example.

[0164] See Figure 9 , is a schematic flow chart of a video processing method provided in an embodiment of the present application, which may include the following steps:

[0165] Step S901: The electronic device 800 responds to a first operation and enters a video recording mode or a video recording process.

[0166] The first operation may be used to instruct the electronic device 800 to enter a video recording mode (or video recording mode) or a video recording process.

[0167] For example, see Figure 10 Schematic diagram of a video recording scenario provided by an embodiment of the present application is shown. A camera application 1020 and icons of multiple other applications are displayed on the main interface of a mobile phone 1010. After a user taps the camera application 1020 icon, the mobile phone 1010 displays a video preview interface 1030 of the camera application 1020 in response to the tap. The video preview interface 1030 displays various shooting modes, including video, photo, portrait, and night scene. Figure 1 The mobile phone 1010 has entered the video recording mode and is in the video recording preview state. In the video recording preview state, the user can enter the video recording process to trigger the mobile phone 1010 by clicking the shooting button 1040.

[0168] exist Figure 10 In the illustrated scenario, the first operation may include a user clicking on the capture button 1040, or may include a user clicking on the camera application 1020 icon and a user clicking on the capture button 1040. In this case, the first operation may instruct the electronic device 800 to enter a video recording process; or, the first operation may include a user clicking on the camera application 1020 icon. In this case, the first operation may instruct the electronic device 800 to enter a video recording mode.

[0169] That is, the electronic device 800 can execute the video processing solution provided by the embodiment of the present application when it enters the video recording mode and is in the video preview state but has not yet entered the video recording state to obtain a video image with a higher dynamic range and clarity. Of course, the electronic device 800 can also execute the video processing solution provided by the embodiment of the present application after entering the video recording process.

[0170] Optionally, the mobile phone 1010 may also provide an HDR video shooting mode, or an HDR mode trigger button. For example, Figure 10The video preview interface 1030 includes an HDR mode trigger button 1060 or an HDR video mode button 1050. When the mobile phone 1010 is in the video preview state or in the video recording process, the user can turn on or off the HDR mode by clicking the HDR mode trigger button 1060. When the HDR mode is turned on, the mobile phone 1010 executes the video processing solution provided in the embodiment of the application to obtain a video image with a higher dynamic range and higher clarity.

[0171] The user can trigger the mobile phone 1010 to enter the HDR video recording mode by clicking the HDR video mode button 1050. The mobile phone 1010 enters the HDR video recording mode, or enters the HDR video recording mode and enters the video recording process, executes the video processing solution provided by the embodiment of the present application, and obtains a video image with higher dynamic range and clarity. In this case, the first operation may include a click operation on the HDR video mode 1050 or the HDR mode trigger button 1060.

[0172] Step S902: The electronic device 800 controls the image sensor to expose each frame alternately according to the first frame exposure parameter and the second frame exposure parameter, and obtains an image pair of each frame output by the image sensor.

[0173] Among them, the first frame exposure parameter is used to indicate that the image sensor outputs an image pair including a first type of exposure image and a second type of exposure image, and the second frame exposure parameter is used to indicate that the image sensor outputs an image pair including a second type of exposure image and a third type of exposure image, or a fourth type of exposure image and a fifth type of exposure image; the exposure levels of the first type of exposure image, the second type of exposure image, and the third type of exposure image are different, or the exposure levels of the first type of exposure image, the second type of exposure image, the fourth type of exposure image, and the fifth type of exposure image are different; the image pair includes two exposure images with different exposure levels.

[0174] It is understandable that the image sensor can output two exposure images with different exposure levels in each frame. In this case, the image sensor can be a sensor with lower capabilities, which can only support outputting two exposure images with different exposure levels in one frame; or it can be a sensor with higher capabilities, in which case, although it can support outputting three exposure images with different exposure levels in one frame (for example, Figure 3 ), but the embodiment of the present application controls it to output two exposure images with different exposure levels in one frame.

[0175] It is worth noting that when the image sensor has low capabilities and can only support the output of two exposure images with different exposure levels in one frame, the embodiment of the present application alternately exposes according to the exposure parameters of the first frame and the exposure parameters of the second frame, so that the image sensor has different exposure strategies in two adjacent frames, thereby achieving at least three exposure images with different exposure levels even when the image sensor has low capabilities.

[0176] In a possible implementation, the first type of exposure image is a long exposure image, the second type of exposure image is a normal exposure image, and the third type of exposure image is a short exposure image. In this case, the first frame exposure parameter can instruct the image sensor to output a long exposure image and a normal exposure image in one frame, and the second frame exposure parameter can instruct the image sensor to output a normal exposure image and a short exposure image in one frame. The exposure time of the long exposure image is greater than the exposure time of the normal exposure image, and the exposure time of the normal exposure image is greater than the exposure time of the short exposure image. For example, Figure 4A As shown, the image sensor outputs exposure L frame and exposure N frame at F1 according to the first frame exposure parameter; and outputs exposure S frame and exposure N frame at F2 according to the second frame exposure parameter.

[0177] In a possible implementation, the first type of exposure image is a normal exposure image, the second type of exposure image is a short exposure image, and the third type of exposure image is a long exposure image. In this case, the first frame exposure parameter may instruct the image sensor to output a short exposure image and a normal exposure image in one frame, and the second frame exposure parameter may instruct the image sensor to output a long exposure image and a short exposure image in one frame. For example, Figure 5A As shown, the image sensor outputs exposure L frame and exposure S frame at F1 according to the second frame exposure parameter; and outputs exposure N frame and exposure S frame at F2 according to the first frame exposure parameter.

[0178] In a possible implementation, the first type of exposure image is a short exposure image, the second type of exposure image is a long exposure image, and the third type of exposure image is a normal exposure image. In this case, the first frame exposure parameter may instruct the image sensor to output a short exposure image and a long exposure image in one frame, and the second frame exposure parameter may instruct the image sensor to output a long exposure image and a normal exposure image in one frame. For example, Figure 5B As shown, the image sensor outputs exposure L frame and exposure S frame at F1 according to the first frame exposure parameter; and outputs exposure N frame and exposure L frame at F2 according to the second frame exposure parameter.

[0179] In one possible implementation, the first frame exposure parameter is used to instruct the image sensor to output the first type of exposure image and the second type of exposure image in one frame, and the second frame exposure parameter is used to instruct the image sensor to output the fourth type of exposure image and the fifth type of exposure image in one frame. At this time, the image sensor outputs exposure images of four different exposure levels in two adjacent frames. Exemplarily, the exposure duration of the first type of exposure image is greater than the exposure duration of the second type of exposure image, the exposure duration of the second type of exposure image is greater than the exposure duration of the fourth type of exposure image, and the exposure duration of the fourth type of exposure image is greater than the exposure duration of the fifth type of exposure image. For example, Figure 6 As shown, the image sensor outputs exposure L frame and exposure N frame at F1 according to the first frame exposure parameter; and outputs exposure S frame and exposure D frame at F2 according to the second frame exposure parameter.

[0180] In a specific application, the electronic device 800 can control the image sensor to expose and output images through output methods such as DOL or stagger.

[0181] In step S903, the electronic device 800 performs multi-frame image fusion processing according to the target exposure image set to obtain an output frame; the target exposure image set includes at least three types of exposure images with different exposure levels, and the at least three types of exposure images with different exposure levels include an image pair of the current frame and at least one exposure image in an image pair of an adjacent frame.

[0182] The target exposure image set may include three, four, five, six, or even more exposure images, but it includes at least three types of exposure images with different exposure levels.

[0183] For example, when the second frame exposure parameter is used to instruct the image sensor to output the second type of exposure image and the third type of exposure image, the target exposure image set may include the first type of exposure image and the second type of exposure image of the current frame, and the third type of exposure image of the adjacent frame; or the first type of exposure image and the second type of exposure image of the current frame, and the second type of exposure image and the third type of exposure image of the adjacent frame. Figure 4A As shown, when the current frame is F1 and the adjacent frame is F2, the target exposure image set of F1 can include the exposure L frame and exposure N frame of F1, and the exposure S frame of F2; or it can include the exposure L frame and exposure N frame of F1, and the exposure N frame of the exposure S frame of F2. Based on the target exposure image set of F1, multi-frame fusion processing can be performed to obtain the output frame 1 of F1. Similarly, based on the target exposure image set of F2, multi-frame fusion processing can be performed to obtain the output frame 2 of F2. And so on, multiple output frames are obtained.

[0184] For example, Figure 4AAs shown, when the current frame is F1, the adjacent frame is F2. At this time, the target exposure image set of F1 can include the exposure L frame and exposure N frame of F1, the exposure S frame and exposure N frame of F2, and the exposure L frame and exposure N frame of F3. The six exposure images in the target exposure image set are subjected to multi-frame fusion processing to obtain the output frame of F1.

[0185] For example, when the second frame exposure parameter is used to instruct the image sensor to output the fourth type of exposure image and the fifth type of exposure image, the target exposure image set may include the first type of exposure image and the second type of exposure image of the current frame, and the fourth type of exposure image and the fifth type of exposure image of the adjacent frame. Figure 6 As shown, when the current frame is F1, the adjacent frame is F2. At this point, the target exposure image set for F1 may include exposure frames L and N of F1, and exposure frames S and D of F2. Multi-frame fusion processing is performed based on the target exposure image set for F1 to obtain output frame 1 of F1. Similarly, multi-frame fusion processing is performed based on the target exposure image set for F2 to obtain output frame 2 of F2. This process is repeated in this way to obtain multiple output frames.

[0186] During the multi-frame fusion process, the electronic device 800 can perform multi-frame fusion processing based on the target exposure image set based on the reference frame of the current frame to obtain the output frame. For example, the electronic device 800 can input each exposure image in the target exposure image set and the target information corresponding to each exposure image into the multi-frame fusion model to obtain the output frame output by the multi-frame fusion model. The target information can refer to Figure 7A and Figure 7B The multi-frame fusion model is an AI model that can be Figure 7A and Figure 7B The network model shown is not limited here.

[0187] Each exposure image in the target exposure image set is a RAW image, and the output frame is a RAW image or an RGB image.

[0188] For each frame, the electronic device 800 obtains a target exposure image set for that frame and performs multi-frame fusion processing based on the target exposure image set to obtain an output frame for that frame. After obtaining the output frame corresponding to each frame, a video file can be generated based on each output frame.

[0189] Optionally, the method may further include step S904, the electronic device 800 exits the video recording in response to the second operation, and obtains a high dynamic range video file, where the high dynamic range video file is generated based on each output frame.

[0190] The second operation may be an operation to instruct the electronic device 800 to exit the video recording mode or the video recording process. Figure 10In the video recording scenario shown, the second operation may include the user clicking the capture button 1040 after the mobile phone 1010 enters the video recording process.

[0191] The embodiment of the present application uses exposure control and fusion algorithms to coordinate processing, so that the exposure strategies of two adjacent frames are different. Even if the image sensor capability is low, two exposure images with different exposure levels can be output in one frame, and more types of exposure images with different exposure levels can be obtained. In addition, at least three types of exposure images with different exposure levels of two adjacent frames are fused into multiple frames, so that more exposure images with different exposure levels can be fused into multiple frames, thereby improving video clarity and dynamic range.

[0192] In one possible implementation, the electronic device 800 may control the image sensor to perform alternating exposure in a row-by-row exposure mode in the current frame according to the first frame exposure parameter, thereby obtaining an image pair output by the image sensor including a first-type exposure image and a second-type exposure image; and control the image sensor to perform alternating exposure in a row-by-row exposure mode in the frame following the current frame according to the second frame exposure parameter, thereby obtaining an image pair output by the image sensor including a second-type exposure image and a third-type exposure image, or a fourth-type exposure image and a fifth-type exposure image. The frame exposure process according to the first frame exposure parameter and the second frame exposure parameter can be referred to. Figure 4C The relevant content will not be repeated here.

[0193] See also Figure 11 The schematic block diagram of the video processing device provided by the embodiment of the present application is shown. The embodiment of the present application further provides a video processing device 1100. The video processing device 1100 may include a video recording module 1110, an exposure control module 1120, an image sensor 1130 and a multi-frame fusion module 1140.

[0194] The video recording module 1110 is configured to respond to a first operation and enter a video recording mode or a video recording process.

[0195] The exposure control module 1120 is used to control the image sensor 1130 to alternately expose each frame according to the first frame exposure parameter and the second frame exposure parameter, and obtain image pairs of each frame output by the image sensor 1130. The first frame exposure parameter is used to instruct the image sensor 1130 to output an image pair including a first type of exposure image and a second type of exposure image, and the second frame exposure parameter is used to instruct the image sensor 1130 to output an image pair including a second type of exposure image and a third type of exposure image, or a fourth type of exposure image and a fifth type of exposure image. The first type of exposure image, the second type of exposure image, and the third type are images with different exposure levels, or the first type of exposure image, the second type of exposure image, the fourth type of exposure image, and the fifth type of exposure image are images with different exposure levels, and the image pair includes two exposure images with different exposure levels.

[0196] The multi-frame fusion module 1140 is used to perform multi-frame image fusion processing based on the target exposure image set to obtain an output frame. The target exposure image set includes at least three types of exposure images with different exposure levels. The at least three types of exposure images with different exposure levels include an image pair of the current frame and at least one exposure image in an image pair of an adjacent frame.

[0197] In one possible implementation, the exposure control module 1120 is specifically used to: control the image sensor to perform alternating exposure in the current frame according to the first frame exposure parameter, and obtain an image pair output by the image sensor including a first type of exposure image and a second type of exposure image; and control the image sensor to perform alternating exposure in the next frame of the current frame according to the second frame exposure parameter, and obtain an image pair output by the image sensor including a second type of exposure image and a third type of exposure image, or a fourth type of exposure image and a fifth type of exposure image.

[0198] In one possible implementation, the first type of exposure image is a long-exposure image, the second type of exposure image is a normal-exposure image, and the third type of exposure image is a short-exposure image; or, the first type of exposure image is a normal-exposure image, the second type of exposure image is a short-exposure image, and the third type of exposure image is a long-exposure image; or, the first type of exposure image is a short-exposure image, the second type of exposure image is a long-exposure image, and the third type of exposure image is a normal-exposure image; wherein the exposure time of the long-exposure image is greater than the exposure time of the normal-exposure image, and the exposure time of the normal-exposure image is greater than the exposure time of the short-exposure image.

[0199] In one possible implementation, the multi-frame fusion module 1140 is specifically used to input each exposure image in the target exposure image set and the target information corresponding to each exposure image into the multi-frame fusion model to obtain an output frame output by the multi-frame fusion model.

[0200] In a possible implementation, when the second frame exposure parameter is used to instruct the image sensor to output the second-category exposure image and the third-category exposure image, the target exposure image set includes the first-category exposure image and the second-category exposure image of the current frame, and the third-category exposure image of an adjacent frame; or the first-category exposure image and the second-category exposure image of the current frame, and the second-category exposure image and the third-category exposure image of an adjacent frame;

[0201] When the second frame exposure parameter is used to instruct the image sensor to output the fourth and fifth exposure images, the target exposure image set includes the first and second exposure images of the current frame, and the fourth and fifth exposure images of the adjacent frame.

[0202] In a possible implementation, each exposure image in the target exposure image set is a RAW image, and the output frame is a RAW image or an RGB image.

[0203] In a possible implementation, the video recording module 1110 is further configured to: exit the video recording in response to the second operation, thereby obtaining a high dynamic range video file, where the high dynamic range video file is generated based on each output frame.

[0204] The above-mentioned video processing device 1100 has the function of implementing the video processing method of the above-mentioned electronic device. This function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function, and the module can be software and / or hardware.

[0205] It should be noted that the information interaction, execution process, etc. between the above-mentioned video processing devices 1100 are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0206] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0207] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process of the above-mentioned method embodiment by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, mobile hard drive, magnetic disk, or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0208] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0209] In the embodiments provided in this application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0211] The electronic device provided in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method as described in any one of the above method embodiments is implemented.

[0212] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0213] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0214] The present application also provides a chip system, comprising a processor coupled to a memory, and executing a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0215] In the above embodiments, the descriptions of each embodiment have different emphases. For portions not described or documented in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. It should be understood that the sequence numbers of the steps in the above embodiments do not imply a sequential order of execution. The execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation of the embodiments of this application. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used solely to distinguish descriptions and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features designated as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of such features. Furthermore, it should be understood that "at least one" in the embodiments of this application includes one or more, where "more" means greater than or equal to two. In the embodiments of this application, "and / or" is simply a description of an association between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. In addition, the character “ / ” in this article generally indicates that the previous and next related objects are in an “or” relationship.

[0216] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized.

[0217] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A video processing method, characterized in that: Applied to an electronic device, the electronic device includes an image sensor, and the method includes: In response to a first operation, entering a video recording mode or a video recording process; controlling the image sensor to alternately expose each frame according to a first frame exposure parameter and a second frame exposure parameter, and obtaining an image pair of each frame output by the image sensor, wherein the first frame exposure parameter is used to instruct the image sensor to output an image pair including a first-category exposure image and a second-category exposure image, and the second frame exposure parameter is used to instruct the image sensor to output an image pair including a second-category exposure image and a third-category exposure image, or a fourth-category exposure image and a fifth-category exposure image, wherein the first-category exposure image, the second-category exposure image, and the third-category exposure image have different exposure levels, or the first-category exposure image, the second-category exposure image, the fourth-category exposure image, and the fifth-category exposure image have different exposure levels, and the image pair includes two exposure images with different exposure levels; Multi-frame image fusion processing is performed according to the target exposure image set to obtain an output frame, wherein the target exposure image set includes at least three types of exposure images with different exposure levels, and the at least three types of exposure images with different exposure levels include an image pair of a current frame and at least one exposure image in an image pair of an adjacent frame.

2. The method according to claim 1, characterized in that Controlling the image sensor to alternately expose each frame according to the first frame exposure parameter and the second frame exposure parameter, and obtaining an image pair of each frame output by the image sensor, comprising: controlling the image sensor to perform alternating exposure in a current frame in an alternating row-by-row exposure manner according to the first frame exposure parameter, to obtain an image pair output by the image sensor including the first-type exposure image and the second-type exposure image; According to the exposure parameters of the second frame, the image sensor is controlled to expose the next frame after the current frame in an exposure mode of alternating row-by-row exposure, so as to obtain an image pair output by the image sensor including the second-category exposure image and the third-category exposure image, or the fourth-category exposure image and the fifth-category exposure image.

3. The method according to claim 1, characterized in that The first type of exposure image is a long exposure image, the second type of exposure image is a normal exposure image, and the third type of exposure image is a short exposure image; or, the first type of exposure image is a normal exposure image, the second type of exposure image is a short exposure image, and the third type of exposure image is a long exposure image; or, the first type of exposure image is a short exposure image, the second type of exposure image is a long exposure image, and the third type of exposure image is a normal exposure image; The exposure time of the long-exposure image is greater than the exposure time of the normal-exposure image, and the exposure time of the normal-exposure image is greater than the exposure time of the short-exposure image.

4. The method according to any one of claims 1 to 3, characterized in that Perform multi-frame image fusion processing based on the target exposure image set to obtain the output frame, including: Each exposure image in the target exposure image set and the target information corresponding to each exposure image are input into a multi-frame fusion model to obtain the output frame output by the multi-frame fusion model.

5. The method according to claim 4, characterized in that When the second frame exposure parameter is used to instruct the image sensor to output the second-category exposure image and the third-category exposure image, the target exposure image set includes the first-category exposure image and the second-category exposure image of the current frame, and the third-category exposure image of the adjacent frame; or the first-category exposure image and the second-category exposure image of the current frame, and the second-category exposure image and the third-category exposure image of the adjacent frame; When the second frame exposure parameter is used to instruct the image sensor to output the fourth category exposure image and the fifth category exposure image, the target exposure image set includes the first category exposure image and the second category exposure image of the current frame, and the fourth category exposure image and the fifth category exposure image of the adjacent frame.

6. The method according to claim 4, characterized in that Each exposure image in the target exposure image set is a RAW image, and the output frame is a RAW image or an RGB image.

7. The method according to claim 1, characterized in that The method further comprises: In response to the second operation, the video recording is exited, and a high dynamic range video file is obtained, where the high dynamic range video file is generated based on each of the output frames.

8. A video processing device, characterized in that: Includes video recording module, exposure control module, image sensor and multi-frame fusion module; The video recording module is used to enter a video recording mode or a video recording process in response to a first operation; The exposure control module is configured to control the image sensor to alternately perform exposure of each frame according to a first frame exposure parameter and a second frame exposure parameter, and obtain an image pair of each frame output by the image sensor, wherein the first frame exposure parameter is used to instruct the image sensor to output an image pair including a first-category exposure image and a second-category exposure image, and the second frame exposure parameter is used to instruct the image sensor to output an image pair including a second-category exposure image and a third-category exposure image, or a fourth-category exposure image and a fifth-category exposure image, wherein the first-category exposure image, the second-category exposure image, and the third-category exposure image have different exposure levels, or the first-category exposure image, the second-category exposure image, the fourth-category exposure image, and the fifth-category exposure image have different exposure levels, and the image pair includes two exposure images with different exposure levels; The multi-frame fusion module is used to perform multi-frame image fusion processing based on a target exposure image set to obtain an output frame. The target exposure image set includes at least three types of exposure images with different exposure levels. The at least three types of exposure images with different exposure levels include an image pair of a current frame and at least one exposure image in an image pair of an adjacent frame.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 7.

Images