Video processing method and device, electronic equipment, medium and program product

Abstract

The invention relates to the technical field of computers, in particular to a video processing method and device, electronic equipment, a medium and a program product. In the method, in the process of generating video picture frames in real time, the electronic equipment can determine whether to perform noise reduction processing of an AIRAW algorithm on RAW data of each video picture frame according to whether a scene of each video picture frame is a low-light scene. If the scene is a low-brightness scene, carrying out noise reduction processing on RAW data of the frame of video picture by adopting an AIRAW algorithm; and if the scene is not the low-brightness scene, performing noise reduction processing on the RAW data of the frame of video picture without adopting the AIRAW algorithm. Thus, in the process that a video needs to be generated in real time, the RAW data collected in the low-light scene can be subjected to noise reduction through the AIRAW algorithm, the problem that the effect of the obtained night scene video is not good enough due to the fact that the equipment loses more image details due to the fact that a YUV noise reduction mode is used is solved, meanwhile, power consumption is saved, and the performance of the electronic equipment is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a video processing method, apparatus, electronic device, medium, and program product. Background Technology

[0002] With the development of technology, users have increasingly higher requirements for the video quality captured by mobile phones and other electronic devices. In particular, the requirements for videos shot at night in low light conditions (hereinafter referred to as "night scene videos") are becoming increasingly demanding.

[0003] Understandably, during the shooting process, the camera's light-sensitive elements (such as image sensors) receive raw data (usually called RAW data) that is converted from light source signals into digital signals. RAW data contains complete image information without any form of compression or color processing.

[0004] Generally, after acquiring raw RAW data and processing it to obtain YUV data (Y represents luminance, and U and V represent chrominance), joint noise reduction processing is performed in the YUV color space, combining information from the time domain and spatial domain. The noise-reduced YUV data is then displayed to improve the quality of night scene videos.

[0005] However, due to the high noise levels in image data captured by image sensors in low-light conditions at night, YUV data loses more image detail compared to RAW data after conversion. Furthermore, due to the unique characteristics of RAW data (e.g., unprocessed color information and high dynamic range), only a subset of traditional spatial domain denoising algorithms are suitable for this data format in software. Therefore, the night scene video quality obtained by current electronic devices is not sufficiently good. Summary of the Invention

[0006] To address the aforementioned issues, this application proposes a video processing method, apparatus, electronic device, medium, and program product. This enables the use of the AIRAW algorithm to reduce noise in low-light scenes when real-time video generation is required. This avoids the problem of insufficient night scene video quality due to the loss of more image details caused by using YUV noise reduction, while also saving power and improving the performance of electronic devices.

[0007] In a first aspect, this application provides a video processing method applied to an electronic device. The method includes: acquiring first RAW data of a video frame N captured by an image sensor of the electronic device during video acquisition; preprocessing the first RAW data of the video frame N to obtain second RAW data of the video frame N; if the shooting scene corresponding to the video frame N belongs to a first scene, using an AI model to perform noise reduction processing on the second RAW data of the video frame N to obtain third RAW data of the video frame N, and obtaining the video frame N based on the third RAW data of the video frame N; if the shooting scene corresponding to the video frame N does not belong to the first scene, not using an AI model to perform noise reduction processing on the second RAW data of the video frame N, and obtaining the video frame N based on the second RAW data of the video frame N.

[0008] Understandably, the first scene can be a low-light scene. If the shooting scene corresponding to the Nth frame of the video belongs to the first scene, the electronic device uses an AI model to perform noise reduction on the second RAW data of the Nth frame to obtain the third RAW data of the Nth frame, and then obtains the Nth frame of the video based on the third RAW data. If the shooting scene corresponding to the Nth frame of the video does not belong to the first scene, the AI ​​model is not used to perform noise reduction on the second RAW data of the Nth frame, and the Nth frame of the video is obtained based on the second RAW data of the Nth frame. In this way, when the electronic device needs to generate a video in real time, it can only use the AI ​​model to perform noise reduction on the RAW data acquired in the first scene, instead of using the AI ​​model for noise reduction in real time, thereby saving power and improving the performance of the electronic device.

[0009] In one possible implementation of the first aspect above, the first scene is a scene whose scene parameters satisfy the first condition, and the scene parameters include brightness values ​​and / or illuminance values.

[0010] It is understandable that luminance and illuminance values ​​can effectively reflect the lighting environment of a scene, and using luminance and / or illuminance values ​​as scene parameters is scientific and reasonable.

[0011] In one possible implementation of the first aspect above, the first condition includes: the luminance value is less than a first luminance threshold, and / or the illuminance value is less than a first illuminance threshold.

[0012] Understandably, the lower the brightness value, the darker the scene; similarly, the lower the illuminance value, the darker the scene. Therefore, when the first condition includes a brightness value less than a first brightness threshold and / or an illuminance value less than a first illuminance threshold, low-brightness environments can be effectively filtered out.

[0013] In one possible implementation of the first aspect above, the shooting scene corresponding to the Nth video frame belongs to the first scene, including: the scene parameters of the Nth video frame satisfy the first condition; and the scene parameters of the Nth video frame are obtained based on the sensitivity information corresponding to the first RAW data of the Nth video frame.

[0014] Understandably, image sensors acquire a variety of raw, unprocessed information when capturing RAW data. Based on the sensitivity information corresponding to the first RAW data of the Nth frame of video, scene parameters can be obtained scientifically and effectively.

[0015] In one possible implementation of the first aspect above, the scene parameters of the Nth frame video are obtained based on the sensitivity information corresponding to the first RAW data of the Nth frame video, including: using an automatic exposure algorithm to calculate the scene parameters based on the sensitivity information corresponding to the first RAW data of the Nth frame video.

[0016] Understandably, automatic exposure algorithms can effectively obtain scene parameters, such as illuminance values.

[0017] In one possible implementation of the first aspect above, the shooting scene corresponding to the Nth video frame belongs to the first scene, including: the scene parameters of the Nth video frame satisfying the first condition; and, the shooting scene corresponding to the (N-1)th video frame belonging to the first scene is obtained based on the YUV data corresponding to the third RAW data of the (N-1)th video frame, and the shooting scene corresponding to the (N-1)th video frame not belonging to the first scene is obtained based on the YUV data corresponding to the second RAW data of the (N-1)th video frame.

[0018] Understandably, since the RAW data of the Nth frame of video is large in volume and has high information content, directly calculating scene parameters representing scene brightness based on the RAW data of the Nth frame of video is resource-intensive. However, YUV data contains less information than RAW data (for example, the YUV image size is smaller than the RAW image size), and YUV data contains sufficient brightness information. Therefore, scene parameters representing scene brightness can be calculated using the YUV data obtained during the generation of the (N-1)th frame of video, greatly saving time and equipment resources.

[0019] In one possible implementation of the first aspect above, the scene parameters of the Nth frame video are obtained based on the YUV data corresponding to the third RAW data of the (N-1)th frame video, including: obtaining the scene parameters of the Nth frame video based on the third RAW data of the (N-1)th frame video by: converting the third RAW data of the (N-1)th frame video into the first YUV data of the (N-1)th frame video; post-processing the first YUV data of the (N-1)th frame video into the second YUV data of the (N-1)th frame video, wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement; calculating the luminance component based on the second YUV data of the (N-1)th frame video, and obtaining the scene parameters of the Nth frame video based on the luminance component.

[0020] Understandably, when noise reduction is required using an AI model (i.e., the AIRAW algorithm) during the generation of the (N-1)th frame of video, the third RAW data of the (N-1)th frame will be obtained. Converting the third RAW data of the (N-1)th frame into the first YUV data of the (N-1)th frame, then post-processing the first YUV data of the (N-1)th frame into the second YUV data of the (N-1)th frame, and calculating the luminance component based on the second YUV data of the (N-1)th frame, and then obtaining the scene parameters of the Nth frame based on the luminance component, is a scientific and reasonable approach.

[0021] In one possible implementation of the first aspect above, the scene parameters of the Nth frame video are obtained based on the YUV data corresponding to the second RAW data of the (N-1)th frame video, including: obtaining the scene parameters of the Nth frame video based on the second RAW data of the (N-1)th frame video by: converting the second RAW data of the (N-1)th frame video into the first YUV data of the (N-1)th frame video; post-processing the first YUV data of the (N-1)th frame video to the second YUV data of the (N-1)th frame video, wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement; calculating the luminance component based on the second YUV data of the (N-1)th frame video, and obtaining the scene parameters of the Nth frame video based on the luminance component.

[0022] Understandably, when it is not necessary to use the AI ​​model (i.e., the AIRAW algorithm) for noise reduction during the generation of the N-1th frame of video, the second RAW data of the N-1th frame of video can be directly converted into the first YUV data of the N-1th frame of video. The first YUV data of the N-1th frame of video can then be post-processed to become the second YUV data of the N-1th frame of video. The luminance component can be calculated based on the second YUV data of the N-1th frame of video, and the scene parameters of the Nth frame of video can be obtained based on the luminance component.

[0023] In one possible implementation of the first aspect above, the second RAW data includes first long frame data and first short frame data, and the third RAW data includes second long frame data and second short frame data; and, when the shooting scene corresponding to the Nth frame video belongs to the first scene, the third RAW data is obtained by using an AI model to perform noise reduction processing on the second RAW data, including: using an AI model to perform noise reduction on the first long frame data to obtain the second long frame data; and directly using the first short frame data as the second short frame data.

[0024] In one possible implementation of the first aspect described above, the video capture process includes a process of shooting video in a first mode, wherein the first mode is a night scene mode or a high dynamic range mode.

[0025] Understandably, shooting in night mode or high dynamic range mode can effectively meet user needs.

[0026] In one possible implementation of the first aspect above, obtaining the Nth frame video image based on the third RAW data of the Nth frame video image includes: converting the third RAW data of the Nth frame video image into the first YUV data of the Nth frame video image; post-processing the first YUV data of the Nth frame video image into the second YUV data of the Nth frame video image, wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement; and obtaining the Nth frame video image based on the second YUV data of the Nth frame video image.

[0027] Understandably, the third RAW data of the Nth frame video is converted into the first YUV data of the Nth frame video, and the first YUV data of the Nth frame video is post-processed into the second YUV data of the Nth frame video. The post-processing includes at least one of hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement. Thus, the Nth frame video can be effectively obtained based on the second YUV data of the Nth frame video.

[0028] Secondly, this application provides a video processing apparatus, comprising: a sensor data acquisition module for acquiring first RAW data of a video frame N acquired by an image sensor during video acquisition; a RAW data preprocessing module for preprocessing the first RAW data of the video frame N to obtain second RAW data of the video frame N; an AIAW algorithm module for using an AI model to perform noise reduction processing on the second RAW data of the video frame N to obtain third RAW data of the video frame N when the shooting scene corresponding to the video frame N belongs to a first scene, and not using an AI model to perform noise reduction processing on the second RAW data of the video frame N when the shooting scene corresponding to the video frame N does not belong to the first scene; and a generation module for obtaining the video frame N based on the third RAW data of the video frame N when the shooting scene corresponding to the video frame N belongs to the first scene, and for obtaining the video frame N based on the second RAW data of the video frame N when the shooting scene corresponding to the video frame N does not belong to the first scene.

[0029] In one possible implementation of the second aspect above, the generation module includes: a YUV data generation module, configured to generate first YUV data of the Nth frame video based on the third RAW data of the Nth frame video when the shooting scene corresponding to the Nth frame video belongs to the first scene, and to generate first YUV data of the Nth frame video based on the second RAW data of the Nth frame video when the shooting scene corresponding to the Nth frame video does not belong to the first scene; a YUV data processing module, configured to post-process the first YUV data of the Nth frame video to the second YUV data of the Nth frame video when the shooting scene corresponding to the Nth frame video belongs to the first scene, and to post-process the first YUV data of the Nth frame video to the second YUV data of the Nth frame video when the shooting scene corresponding to the Nth frame video does not belong to the first scene; wherein, the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement; and a video frame generation module, configured to obtain the Nth frame video based on the second YUV data of the Nth frame video.

[0030] Thirdly, this application provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the video processing method provided in the first aspect and various possible implementations described above.

[0031] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the video processing method provided in the first aspect and various possible implementations described above.

[0032] Fifthly, this application provides a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the video processing method provided in the first aspect and various possible implementations described above.

[0033] Understandably, the beneficial effects of the second to fifth aspects mentioned above refer to the first aspect and various possible implementations, which will not be elaborated here. Attached Figure Description

[0034] Figure 1A According to some embodiments of this application, a schematic diagram of a video frame 001 before a mobile phone 100 starts recording is shown;

[0035] Figure 1B According to some embodiments of this application, a schematic diagram of a video screen 002 of a mobile phone 100 is shown;

[0036] Figure 1C According to some embodiments of this application, a schematic diagram of a video frame 003 during the recording process of a mobile phone 100 is shown;

[0037] Figure 2A According to some embodiments of this application, another schematic diagram of a video frame 004 before a mobile phone 100 starts recording is shown;

[0038] Figure 2B According to some embodiments of this application, another schematic diagram of a video frame 005 during the recording process of a mobile phone 100 is shown;

[0039] Figure 3 According to some embodiments of this application, a schematic diagram of a video processing apparatus 20 is shown;

[0040] Figure 4 According to some embodiments of this application, a schematic diagram of a data stream for displaying the Nth frame of video based on a video processing device 20 is shown;

[0041] Figure 5A A schematic diagram of a video processing apparatus 30 is shown according to some embodiments of this application;

[0042] Figure 5B According to some embodiments of this application, a schematic diagram of a video processing apparatus 40 is shown;

[0043] Figure 6According to some embodiments of this application, a schematic diagram of a video processing method is shown;

[0044] Figure 7 According to an embodiment of this application, a schematic diagram of another video processing method is shown;

[0045] Figure 8 According to an embodiment of this application, a schematic diagram of a video processing apparatus 50 is shown;

[0046] Figure 9 According to an embodiment of this application, a schematic diagram of the hardware structure of a mobile phone 100 is shown. Detailed Implementation

[0047] The illustrative embodiments of this application include, but are not limited to, a video processing method, apparatus, electronic device, medium, and program product.

[0048] The following is a brief introduction to some of the technical terms used in this application.

[0049] (1) RAW data: refers to raw image data that is neither compressed nor encoded. In digital image processing, RAW data usually refers to the raw pixel values ​​obtained directly from the image sensor. For example, the image sensor can be a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) of a camera device.

[0050] (2) YUV Data: YUV data is a color space representation method used to represent color in video and image processing. It divides the color information of an image into two parts: luminance (Y) and chrominance (U and V). Y represents luminance information, that is, the brightness of the image, short for luminance. The Y component contains all the information of a black and white image, and it determines the contrast and detail of the image. U: Represents the blue color difference component, that is, the difference between the blue component and the luminance. V: Represents the red color difference component, that is, the difference between the red component and the luminance.

[0051] The embodiments of this application will be described in detail below.

[0052] As described in the background section, during night scene video shooting, RAW data is typically processed to obtain YUV data. However, YUV data loses more image details compared to RAW data. Therefore, even with noise reduction processing on the YUV data, the resulting night scene video is not of sufficiently good quality.

[0053] For example, Figures 1A to 1CAccording to some embodiments of this application, a schematic diagram of a process for shooting video in low-light conditions at night is shown. Specifically, refer to... Figure 1A The video preview screen 001 shown on the mobile phone 100 is displayed after the user operates the "Video Recording" function control K1 in the camera application of the mobile phone 100. The mobile phone 100 detects the video preview command and displays the video preview screen 001 before recording begins. When the user clicks the "Night Scene Recording" control on the video preview screen 001, the mobile phone 100 detects that the recording function is used and the night scene mode is enabled. At this time, the camera's image sensor obtains RAW data and processes it into YUV data, then uses noise reduction on the YUV data to improve image quality. Furthermore, the image is displayed based on the noise-reduced YUV data. Figure 1B The preview video frame 002 shown is shown below. After the user clicks the "Start Recording" control, the phone 100 will begin recording the video and generate a preview video frame during the recording process, as shown below. Figure 1C The video footage shown is 003, depicting the recording process.

[0054] However, in backlit conditions at night, the light is dim, and the image data collected by the image sensor contains a lot of noise. Furthermore, the YUV data loses a lot of detail, making the video image previewed (i.e., displayed) before or during recording too dark, resulting in blurry figures in the video image seen by the user.

[0055] Understandably, with the development of artificial intelligence (AI) technology, AI technology can be applied to noise reduction processing of RAW data. By combining AI technology with RAW data, noise in the video can be effectively reduced while retaining more image details, thus improving image quality.

[0056] Therefore, to address the aforementioned issues, this application proposes a video processing method that pre-learns the mapping relationship between noisy RAW data (i.e., noisy images) and clean RAW data (i.e., clean images) in various low-light scenes using artificial intelligence techniques (e.g., AI models). When real-time video frame generation is required, during the generation of each video frame, after acquiring the raw RAW data from the image sensor and performing preliminary preprocessing, the learned AI technology is used to denoise the preprocessed RAW data. Then, the denoised RAW data is converted into YUV data, and the video frame is displayed based on the generated YUV data.

[0057] It's understandable that noise reduction is performed on images in RAW format, and since RAW data contains more information, the quality of the displayed video frames is better.

[0058] For example, Figure 2A as well as Figure 2B The following are shown in relation to Figures 1A to 1C In the same recording scenario, the video frames 004 (before recording) and 005 (during recording) were obtained by using AI technology to reduce noise in RAW data. Understandably, because RAW data contains more information, the facial features of the people in video frames 004 and 005 are clearer than those in video frames 002 and 003, especially in low-light conditions. For example, while the face is not visible in the dashed frame 002A of video frame 002, the white mask worn by the person is visible in the dashed frame 004A of video frame 004. This allows users to obtain better results when shooting with a mobile phone.

[0059] For ease of explanation, the algorithm that uses AI technology (e.g., AI models) to denoise RAW data will be referred to as the "AIRAW algorithm" below.

[0060] Furthermore, during video recording, the lighting conditions can change as the user moves or over time. The lighting environment of the raw RAW data captured by the phone may not correspond to the low-light scene. If the AIRAW algorithm is used to denoise the RAW data when generating each video frame, it will consume a lot of computing resources.

[0061] Therefore, in some embodiments, during the real-time generation of video frames, the electronic device can determine whether to perform AIRAW algorithm noise reduction on the RAW data of each video frame based on whether the scene of each video frame is a low-light scene. Specifically, if the scene of a video frame is a low-light scene, the electronic device can perform noise reduction on the RAW data of that frame using the AIRAW algorithm (e.g., an AI model); if the scene of a video frame is not a low-light scene, the electronic device can choose not to perform noise reduction on the RAW data of that frame using the AIRAW algorithm. Thus, when the electronic device needs to generate a video in real time, it can only use the AIRAW algorithm for noise reduction on the RAW data acquired in a low-light scene, and not use the AIRAW algorithm for the RAW data acquired in a non-low-light scene, thereby saving power consumption and improving the performance of the electronic device.

[0062] For example, when a user opens the camera app on their phone and uses the video recording function, after enabling night mode for backlight recording, even though it's currently in "night mode," the AIRAW algorithm can be dynamically adjusted based on the brightness of the scene corresponding to the current video frame. When the current video frame is determined to be in a bright environment, the noise in the raw RAW data acquired is relatively low, and the preprocessed RAW data can be directly converted to YUV data so that the current video frame can be displayed based on the YUV data.

[0063] Understandably, in this embodiment, when the recording function is enabled, the electronic device can use the video processing method proposed in this embodiment to display video frames in real time during the preview process before detecting the user clicking the "Start Recording" control. Alternatively, the electronic device can use the AIRAW algorithm to process the RAW data and display video frames in real time during the recording process after detecting the user clicking the "Start Recording" control.

[0064] Understandably, the AIRAW algorithm used by electronic devices to reduce noise in RAW data can be enabled in modes beyond those mentioned above, such as the recording function and "Night Mode" in camera applications. This means that when recording in HDR mode, electronic devices can also use AI technology to reduce noise in RAW data, resulting in better video quality.

[0065] It is understood that the embodiments of this application can be applied to any electronic device with a camera device, including but not limited to mobile phones, tablets, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), servers, server clusters, etc., without limitation.

[0066] Figure 3An embodiment of this application illustrates a video processing apparatus 20. Specifically, the video processing apparatus 20 includes a sensor node 200, a thin front end (TFE) node 201, a global motion estimation (GME) node 202, an AIRAW algorithm node 203, an offline front end (OFE) node 204, an image-processing engine (IPE) node 205, a preview and recording processing node 206, a preview node 207, and a video file generation node 208.

[0067] Specifically, sensor node 200 can function as a sensor data acquisition module, used to acquire data from the scene the user wants to capture, obtaining RAW data Q1. RAW data Q1 is the raw pixel value directly obtained from the image sensor. It can be a single image or multiple images at different exposure levels. For example, it could include long-frame RAW data with long exposure times and short-frame RAW data with short exposure times.

[0068] The thin front-end node 201 serves as a RAW data preprocessing module, used to preprocess the obtained RAW data Q1 to obtain RAW data Q2. Preprocessing may include at least one of the following operations: pre-distortion pre-correction (PDPC), adaptive Bayer filter (ABF), channel gain, lens shading correction (LSC), and white balance (WB). In some embodiments, RAW data Q1 typically exists in Bayer format, i.e., color filter array (CFA) data. It is understood that RAW data preprocessing of RAW data Q1 is essentially preprocessing a Bayer format of Bayer format. Pre-distortion correction is used to correct lens-induced distortions, such as barrel or pincushion distortion. Adaptive Bayer filter is used for noise reduction in the initial stage of processing, thereby preventing noise amplification. Channel gain is used to adjust the gain of different color channels to ensure color balance and accuracy. Lens shading correction: Used to correct uneven image brightness and color inconsistencies caused by lens optical characteristics and other factors. White balance: Adjusts the white balance of the image to ensure color accuracy under different lighting conditions.

[0069] The global motion estimation node 202 can function as a correction information acquisition module, passing the RAW data Q2 to the AIRAW algorithm node 203; it can also output correction information based on the RAW data Q2 to assist in image correction. For example, the correction information can be a lens distortion correction (LDC) matrix. Understandably, the output LDC matrix can be used for subsequent processing of the YUV data or for other processing.

[0070] AIRAW algorithm node 203 can be used as an AIRAW algorithm module to perform noise reduction on RAW data Q2 using the AIRAW algorithm to obtain RAW data Q3. Specifically, as shown... Figure 3 As shown, the AIRAW algorithm node 203 includes a source plugin, the AIRAW algorithm, and a sink plugin. The source plugin is used to receive the data input to the AIRAW algorithm node 203, the AIRAW algorithm is used to process the received data using AI technology, and the sink plugin is used to output the data processed by the AIRAW algorithm. In some embodiments, the AIRAW algorithm node 203 denoises the RAW data Q2 based on the trained AI model for denoising to obtain the RAW data Q3. In some implementations, the architecture of the AI ​​model used can be the existing U-Net, ResNet, etc., and the specific model architecture is not required here. For example, an AI model that can be used for denoising can be obtained in the following ways: (1) Collect a large amount of RAW data in advance, including noisy and corresponding clean (no noise or less noise) image pairs, for training the model. Among them, the noisy RAW data is the pre-processed sample. (2) Define a loss function, for example, to measure the difference between the model prediction result and the actual result. For example, commonly used loss functions include mean squared error, structural similarity index, etc. (3) Input noisy RAW data samples into the model, update the model parameters through the backpropagation algorithm, calculate the loss function, and output the model when the loss function is less than the loss function threshold.

[0071] Offline front-end node 204 can serve as a YUV data generation module, used to generate YUV data P1 from RAW data Q3. For example, when RAW data Q3 includes multiple RAW image data, the specific operations for generating YUV data P1 from RAW data Q3 can include the following hardware processing flow: anchor point processing, HDR fusion, tone mapping, and color space conversion. Anchor point processing is used to select one or more frames from RAW data Q3 as reference frames (i.e., anchor frames). HDR fusion is used to merge multiple RAW data (images) with different exposures into a single high dynamic range image. Tone mapping is used to map the high dynamic range of RAW data to a suitable dynamic range. Color space conversion is used to convert RAW data into YUV data.

[0072] Image processing engine node 205 serves as a YUV data processing module, used for post-processing (i.e., YUV data processing) based on YUV data P1 to generate YUV data P2. Specifically, image processing engine node 205 may include a noise processing system and a pixel processing system. The noise processing system is used to reduce image noise, and the pixel processing system is used to process each pixel of the image. Specifically, image processing engine node 205 is used to perform hardware denoising (such as multi-frame noise reduction (MFNR) and multi-frame super-resolution (MFSR)), resizing, color correction (such as chromatic aberration correction and chromaticity suppression), and detail enhancement (such as skin tone enhancement) on YUV data P1. Understandably, when the image processing engine node 205 performs post-processing on the YUV data P1, it can use the correction information (such as the lens distortion correction matrix) output by the global motion estimation node 202 to perform post-processing on the YUV data P1. For example, it can use the correction information to resize the YUV data P1. Alternatively, it can process the YUV data P1 without using the correction information, which is not required here.

[0073] The preview and recording processing node 206 is used to generate video frames for preview or recording files based on YUV data P2. Preview refers to the process of displaying video frames before recording after the camera application enables the recording function, or displaying video frames during the recording process. It is understood that during recording, at least two streams of video frame data are required: one for display (i.e., preview) and the other for saving as a recording file. In this embodiment, the preview and recording processing node 206 is used to process the YUV data P2 into the video frame data required for previewing, and also to process the YUV data P2 into the video frame data required for generating a recording file.

[0074] Preview 207 is used to display video frames. For example, it converts the generated video frames for display into data that the electronic device can support for display.

[0075] Generate video recording file 208, which is used to save the video footage data of the corresponding recording file. For example, data can be saved during the recording process.

[0076] It is understood that in some other embodiments, the video processing apparatus 20 may not include the global motion estimation node 202 described above.

[0077] Understandably, an image sensor can capture video of the current scene, thereby acquiring RAW data needed to generate the Nth frame of video. It is also understandable that the RAW data of the Nth frame of video captured by the image sensor is uncompressed and unencoded raw image data. Furthermore, during video capture, multiple frames of RAW data can be acquired for the same scene.

[0078] For example, during video shooting, both long-frame and short-frame images with longer exposure times can be captured for the same scene. That is, the RAW data Q1 corresponding to the Nth frame of the video can include both long-frame and short-frame images. It is understood that the long-frame images (hereinafter referred to as "long-frame RAW data") and short-frame images (hereinafter referred to as "short-frame RAW data") for the same scene in RAW data Q1 can be combined to generate the same video frame.

[0079] Understandably, when recording video of the same scene, the long and short frames used to generate the same video frame can be image data at different exposure levels captured simultaneously, or they can be rapidly captured consecutively by the image sensor at very close intervals. The capture interval is usually very short. Although the long and short frames are not captured simultaneously in a strictly sense, due to the short capture interval, they can generally be considered to represent the same scene in the same video frame.

[0080] Understandably, RAW data Q1 includes both long-frame and short-frame RAW data. In this case, the AIRAW algorithm can be used for noise reduction on both long-frame and short-frame RAW data, or the AIRAW algorithm can be used for noise reduction only on long-frame RAW data without noise reduction on short-frame RAW data.

[0081] To make it easier to understand, the following will be combined with... Figure 4 Specifically, the video processing device 20 processes a long-frame RAW data and a short-frame RAW data of the same scene collected in real time to generate a video frame, as shown in the data stream diagram.

[0082] Figure 4 According to an embodiment of this application, a data stream diagram of the Nth frame of a video image displayed by a video processing device 20 is shown. The example described uses AIRAW noise reduction processing on long-frame RAW data, while short-frame RAW data is not subjected to AIRAW noise reduction processing.

[0083] Specifically, sensor node 200 acquires one long frame RAW data L1 (i.e., ...) of the Nth frame of the corresponding video. Figure 4 The middle part consists of long frame L1 and one short frame RAW data M1 (i.e., Figure 4 The middle frame is short frame M1.

[0084] The thin front-end node 201 preprocesses the long frame RAW data L1 to obtain the long frame RAW data L2 (i.e. Figure 4 The middle frame is the long frame L2); the short frame RAW data M1 is preprocessed to obtain the short frame RAW data M2 (i.e., Figure 4 (Middle is the short frame M2).

[0085] The global motion estimation node 202 transmits the long-frame RAW data L2 to the AIRAW algorithm node 203 and the short-frame RAW data M2 to the offline front-end node 204. Furthermore, the global motion estimation node 202 can also obtain the correction information for the corresponding long-frame RAW data L2 and the correction information for the corresponding short-frame RAW data M2, which can be used for calculations in other aspects.

[0086] AIRAW algorithm node 203 uses a trained AI model to denoise the long frame RAW data L2, resulting in long frame RAW data L3.

[0087] The offline front-end node 204 fuses the long frame RAW data L3 and the short frame RAW data M2 to generate a YUV data (image), resulting in YUV data P1.

[0088] Furthermore, image processing engine node 205 performs post-processing on YUV data P1 to obtain YUV data P2. After preview and recording processing node 206 generates video frame data for previewing based on YUV data P2, preview 207 displays the Nth frame of video. After preview and recording processing node 206 generates video frame data to be saved in the recording file based on YUV data P2 obtained from image processing engine node 205, recording file generation 208 saves the video frame as a recording file.

[0089] As mentioned earlier, when real-time generation of video frames is required, the scene corresponding to the Nth video frame can be used to determine whether the scene is a low-brightness scene. In this case, the AIRAW algorithm can be omitted to reduce noise in the RAW data, thereby saving power consumption and improving the performance of electronic devices.

[0090] In some embodiments, scene parameter values ​​(e.g., illuminance value, luminance value) representing the scene brightness can be calculated based on the RAW data of the acquired Nth frame of video. This determines whether the scene corresponding to the Nth frame of video is a low-brightness scene, and consequently determines whether noise reduction processing using the AIRAW algorithm is needed when generating the Nth frame of video. Specifically, the auto exposure algorithm (AE) can be used to calculate the scene parameters based on the ISO (International Organization for Standardization) information obtained when the image sensor acquires RAW data.

[0091] In other embodiments, since the RAW data of the Nth frame of video is large in volume and contains a high amount of information, directly calculating scene parameters representing scene brightness based on the RAW data of the Nth frame is resource-intensive. YUV data, on the other hand, contains less information than RAW data (e.g., the YUV image size is smaller than the RAW image size) and contains sufficient brightness information. Therefore, scene parameters representing scene brightness can be calculated using the YUV data obtained during the generation of the (N-1)th frame of video. Furthermore, since the time interval between video frames is short during video generation—that is, the time interval corresponding to the RAW data collected for generating different video frames is short—the scene parameters calculated based on the YUV data obtained during the generation of the previous video frame can approximate the current scene brightness. Therefore, it is scientifically reasonable to determine whether to use the AIRAW algorithm for noise reduction on the preprocessed RAW data. Understandably, the scene parameter values ​​calculated from the YUV data obtained during the generation of the previous video frame are used to determine whether the current scene meets the scene conditions (e.g., whether it is a low-brightness scene), which greatly saves time and device resources.

[0092] Specifically, when generating the Nth frame of video, scene parameter values ​​(e.g., illuminance and brightness values) can be calculated using the processed YUV data (e.g., YUV data P2) obtained during the generation of the Nth (N is an integer greater than or equal to 1) frame of video. And when it is determined that the calculated scene parameter values ​​meet the scene parameter conditions, it is determined during the generation of the Nth frame of video that the preprocessed RAW data needs to be processed using the AIRAW algorithm.

[0093] For example, by default, during the generation of the initial video frame, the AIRAW algorithm is used to denoise the preprocessed RAW data. Furthermore, during the generation of each video frame, the processed YUV data is used to calculate scene parameter values, which serve as the basis for determining whether further processing of the AIRAW data is needed when generating the next video frame.

[0094] Understandably, illuminance values ​​reflect the total amount of light illuminating an object's surface in a scene. In a low-brightness scene, the light source provides less light, resulting in a reduced luminous flux reaching the object's surface, hence a low illuminance value. When the scene parameter value is the illuminance value, the higher the calculated illuminance value, the brighter the current scene. In some embodiments, if the illuminance value is less than a first illuminance threshold, the scene parameter conditions are considered met.

[0095] Brightness values ​​can be determined by an object's own luminescence or reflection characteristics. In low-light scenes, the illuminance received by an object's surface is low, resulting in less reflected light and a lower brightness value. In some embodiments, a scene parameter condition is considered met when the brightness value is less than a first brightness threshold.

[0096] Figure 5A An embodiment of this application illustrates a video processing apparatus 30. The apparatus is described using the determination of scene parameters based on photosensitivity information obtained when acquiring RAW data from an image sensor as an example. Specifically, the video processing apparatus 30 includes a sensor node 300, a thin front-end node 301, a global motion estimation node 302, an AIRAW algorithm node 303, an offline front-end node 304, an image processing engine node 305, a preview and recording processing node 306, a preview node 307, a video file generation node 308, an automatic exposure algorithm node 310, and an information statistics node 309.

[0097] It is understandable that sensor node 300 has the same function as sensor node 200, thin front-end node 301 has the same function as thin front-end node 201, global motion estimation node 302 has the same function as global motion estimation node 202, offline front-end node 304 has the same function as offline front-end node 204, image processing engine node 305 has the same function as image processing engine node 205, preview and recording processing node 306 has the same function as preview and recording processing node 206, preview 307 has the same function as preview 207, and generating recording file 308 has the same function as generating recording file 208. Further details are omitted here.

[0098] The automatic exposure algorithm node 310 calculates scene parameter values ​​(e.g., illuminance values) using an existing automatic exposure algorithm based on the photosensitivity information obtained when the sensor node 300 acquires RAW data and the average brightness transmitted by the sensor node 300 (e.g., the average brightness obtained from the previous RAW data acquired by the corresponding sensor node 300), and then transmits the scene parameters to the AIRAW algorithm node 303. The automatic exposure algorithm node 310 also transmits the processed YUV data to the information statistics node 309.

[0099] The information statistics node 309 is used to collect and store parameter information about the sensor and environmental parameters. For example, the information statistics node 309 can save the scene parameter information calculated by the automatic exposure algorithm node 310 and pass the acquired scene parameter information to the AIRAW algorithm node 303. In addition, it can also receive data transmitted by the thin front-end node 301 and pass it to the global motion estimation node 302.

[0100] AIRAW algorithm node 303 serves as another AIRAW algorithm module, possessing the functions of AIRAW algorithm node 203 and also determining whether the received preprocessed RAW data requires AIRAW algorithm noise reduction. If it is determined that AIRAW algorithm noise reduction is not required, the received preprocessed RAW data is passed to offline front-end node 304; if it is determined that AIRAW algorithm noise reduction is required, the received preprocessed RAW data is processed using the AIRAW algorithm. In some embodiments, the scene parameters received from information statistics node 309 can be used to determine whether the received preprocessed RAW data requires AIRAW algorithm noise reduction. If the shooting scene corresponding to the current video frame is a low-light scene (e.g., when the scene parameters meet the scene parameter conditions), the preprocessed RAW data is denoised using the AIRAW algorithm (e.g., an AI model) to obtain denoised RAW data; if the shooting scene corresponding to the current video frame is not a low-light scene, the AIRAW algorithm (e.g., an AI model) is not used to denoise the processed RAW data. Similarly, the AIRAW algorithm node 303 includes a source plugin, the AIRAW algorithm, and a slot plugin. The source plugin receives data input to the AIRAW algorithm node 303, the AIRAW algorithm processes the received data for the corresponding low-brightness scene using AI technology, and the slot plugin outputs the data processed by the AIRAW algorithm.

[0101] The following describes the data flow between modules of the video processing device 30 when processing RAW data corresponding to the scene corresponding to the initial video frame in the current field during video processing, based on the embodiments of this application.

[0102] Specifically, sensor node 300 acquires RAW data J1 of the current scene (i.e., the scene corresponding to the Nth video frame). Then, thin front-end node 301 preprocesses RAW data J1 to obtain RAW data J2. RAW data J2 is then passed to global motion estimation node 302 via information statistics node 309. Global motion estimation node 302 calculates correction information based on RAW data J2 for other calculations (e.g., to assist image processing engine node 305 in calculations); and also passes RAW data J2 to AIRAW algorithm node 303. Furthermore, during the acquisition of RAW data J1, sensor node 300 passes the photosensitivity information and average brightness information corresponding to RAW data J1 to automatic exposure algorithm node 310. Automatic exposure algorithm node 310 calculates scene parameter values ​​based on the photosensitivity information and average brightness information of the corresponding RAW data J1 and passes them to information statistics node 309. Information statistics node 309 then passes the calculated scene parameters to AIRAW algorithm node 303.

[0103] The AIRAW algorithm node 303 determines whether noise reduction processing is needed for RAW data J2 based on the scene parameters received from the information statistics node 309. If the current scene parameters meet the scene parameter conditions, it determines that the current scene is a low-brightness scene and that noise reduction processing is needed for RAW data J2, generating RAW data J3 and passing it to the offline front-end node 304. If the current scene parameters do not meet the scene parameter conditions, it determines that the current scene is not a low-brightness scene and that noise reduction processing is not needed for RAW data J2, directly passing it to the offline front-end node 304 (not shown in the figure).

[0104] Offline front-end node 304 generates YUV data V1 based on RAW data J3. Image processing engine node 305 processes YUV data V1 to obtain YUV data V2. Preview and recording processing node 306 generates video frames for previewing and saving as recording files based on YUV data V2. Preview 307 displays the video frames. Recording file generation 308 saves the video frame data as a recording file. Similarly, the subsequent processing of offline front-end node 304 after receiving RAW data J2 is the same and will not be described in detail here.

[0105] Figure 5BAn embodiment of this application illustrates a video processing apparatus 40. The apparatus is described using the example of determining scene parameters based on YUV data obtained during the generation of the previous video frame. Specifically, the video processing apparatus 40 includes a sensor node 400, a thin front-end node 401, a global motion estimation node 402, an AIRAW algorithm node 403, an offline front-end node 404, an image processing engine node 405, a preview and recording processing node 406, a preview node 407, a video file generation node 408, and a perception engine 409.

[0106] It is understandable that sensor node 400 has the same function as sensor node 200, thin front-end node 401 has the same function as thin front-end node 201, global motion estimation node 402 has the same function as global motion estimation node 202, offline front-end node 404 has the same function as offline front-end node 204, image processing engine node 405 has the same function as image processing engine node 205, preview and recording processing node 406 has the same function as preview and recording processing node 206, preview 407 has the same function as preview 207, and generating recording file 408 has the same function as generating recording file 208. Further details are omitted here.

[0107] The perception engine 409 is used to calculate scene parameter values ​​based on the processed YUV data obtained from the image processing engine node 405, and feed the scene parameters back to the AIRAW algorithm node 403. It is also used to directly transmit the processed YUV data to the preview and recording processing node 406.

[0108] AIRAW algorithm node 403 serves as another AIRAW algorithm module, possessing the functions of AIRAW algorithm node 203 and also determining whether the received preprocessed RAW data requires AIRAW algorithm denoising. If it is determined that AIRAW algorithm denoising is not needed, the received preprocessed RAW data is passed to offline front-end node 404; if it is determined that AIRAW algorithm denoising is needed, the received preprocessed RAW data is processed using the AIRAW algorithm, and the denoised RAW data is passed to offline front-end node 404. Specifically, AIRAW algorithm node 403 can determine whether the received preprocessed RAW data needs AIRAW algorithm denoising based on scene parameters received from perception engine 409. Understandably, the processing procedure of AIRAW algorithm node 403 on the received preprocessed RAW data based on scene parameters is essentially the same as that of AIRAW algorithm node 303 described above, and will not be elaborated upon here.

[0109] The following describes the data flow between modules of the video processing device 40 when processing RAW data corresponding to the scene corresponding to the initial video frame in the current field during video processing, based on the embodiments of this application.

[0110] Specifically, sensor node 400 acquires RAW data T1 of the current scene (i.e., the scene of the initial video frame). Then, thin front-end node 401 preprocesses RAW data T1 to obtain RAW data T2. Global motion estimation node 402 calculates correction information based on RAW data T2 for other calculations (e.g., to assist image processing engine node 405 in calculations). RAW data T2 is then passed to AIRAW algorithm node 403. AIRAW algorithm node 403 performs noise reduction processing on RAW data T2 to generate RAW data T3. Offline front-end node 404 generates YUV data K1 based on RAW data T3. Image processing engine node 405 processes YUV data K1 to obtain YUV data K2. Preview and recording processing node 406 generates video frames for preview and saving as recording files based on YUV data K2. Preview 407 displays the video frames. Recording file generation 408 saves the video frames as recording files.

[0111] Furthermore, the perception engine 409 calculates the scene parameter values ​​corresponding to the initial video frame based on the YUV data K2 and sends them to the AIRAW algorithm node 403 for determining whether to use the AIRAW algorithm for noise reduction processing in the next video frame.

[0112] The following describes, based on embodiments of this application, the data flow between modules when the video processing device 40 processes the RAW data corresponding to the current scene (the scene of the Nth frame of video, where the Nth frame is not the initial video frame) during video processing. Furthermore, the description will take the example of the AIRAW algorithm node 403 obtaining the scene parameter values ​​corresponding to the scene of the previous frame (the (N-1)th frame of video).

[0113] Specifically, sensor node 400 acquires RAW data T1 of the current scene (e.g., the scene corresponding to the Nth frame of video). Then, thin front-end node 401 preprocesses RAW data T1 to obtain RAW data T2. Global motion estimation node 402 calculates correction information based on RAW data T2 for other calculations (e.g., to assist image processing engine node 405 in calculations). RAW data T2 is then passed to AIRAW algorithm node 403. AIRAW algorithm node 403 determines whether to apply the AIRAW algorithm for noise reduction to RAW data T2 based on scene parameters received from image processing engine node 405 (e.g., scene parameters of the (N-1)th frame of video). If the scene parameter values ​​meet the scene parameter conditions, the preprocessed RAW data is denoised using the AIRAW algorithm; otherwise, no noise reduction is applied.

[0114] When noise reduction is required, AIRAW algorithm node 403 performs noise reduction on RAW data T2 to obtain RAW data T3. AIRAW algorithm node 403 sends RAW data T3 to offline front-end node 404, and the YUV data generation module generates YUV data K1 based on RAW data T3. Image processing engine node 405 processes YUV data K1 to obtain YUV data K2. Perception engine 409 obtains scene parameters based on YUV data K2 and sends them to AIRAW algorithm node 403 for scene determination in the next video frame (the N+1th video frame). Perception engine 409 also passes YUV data K2 to preview and recording processing module 406. Preview and recording processing module 406 generates video frames based on YUV data K2. Preview 407 displays the Nth video frame and generates and saves a recording file 208 based on the generated video frames.

[0115] When noise reduction is not required, the YUV data generation module generates YUV data K1 based on the RAW data T2. The image processing engine node 405 processes the YUV data K1 to obtain YUV data K2. Preview 207 generates and displays video frames based on the YUV data K2. Recording file generation 208 generates and saves the data corresponding to the video frames based on the YUV data K2.

[0116] Figure 6 According to an embodiment of this application, a schematic diagram of a video processing method is shown. During this video processing method, the decision to perform AIRAW calculation is made based on whether the current video frame depicts a low-light scene. The executing device can be any device with camera hardware and camera applications, such as the aforementioned mobile phone 100. The specific steps are as follows:

[0117] S601, a first video recording command corresponding to the first shooting mode is detected.

[0118] In some embodiments, the first video recording instruction includes a preview instruction before video recording or a video recording start instruction. The first recording mode may include a night mode or an HDR mode.

[0119] For example, when a user selects the "record" control in the camera application and enables night mode or HDR mode, the phone 100 detects the recording function and the activation of the first mode. At this time, it will enter the preview process before video recording for the corresponding first mode, that is, it detects the preview instruction before video recording for the corresponding first shooting mode.

[0120] For example, after a user selects the "record" control in the camera application and enables night mode or HDR mode, the user then selects the "start recording" control in the camera application. At this point, the phone 100 detects the start recording operation and enters the recording process, that is, it detects the video start recording command corresponding to the first shooting mode.

[0121] S602, acquires RAW data of the Nth frame of video.

[0122] In some embodiments, the mobile phone 100 can capture video of the current scene using an image sensor, thereby acquiring RAW data of the Nth frame of video.

[0123] Understandably, after acquiring the RAW data of the Nth frame of video, the mobile phone 100 can preprocess the RAW data of the Nth frame to obtain preprocessed RAW data. This preprocessing can include at least one of the following operations: pre-distortion correction, adaptive Bayer filtering, channel gain adjustment, lens shading correction, and white balance.

[0124] S603, determine whether the scene of the Nth frame of the video is a low-brightness scene.

[0125] After acquiring the RAW data of the Nth frame of video, mobile phone 100 can determine whether the scene of the Nth frame is a low-brightness scene. If the scene of the Nth frame is a low-brightness scene, it proceeds to S604 and uses the AIRAW algorithm for noise reduction processing; if the scene of the Nth frame is not a low-brightness scene, it proceeds to S606 and does not use the AIRAW algorithm for noise reduction processing.

[0126] Specifically, the mobile phone 100 can use scene parameter values ​​(e.g., illuminance value, brightness value) that can characterize the scene of the current Nth frame of video. When the scene parameter values ​​that can characterize the scene of the current Nth frame of video meet the scene parameter conditions (e.g., illuminance value is less than a first illuminance threshold, brightness value is less than a first brightness threshold), the scene corresponding to the Nth frame of video is determined to be a low-brightness scene, and the process proceeds to S604; otherwise, the process proceeds to S606.

[0127] In some embodiments, after acquiring the RAW data of the Nth frame of video, the mobile phone 100 can preprocess the RAW data of the Nth frame to obtain preprocessed RAW data of the Nth frame. Then, based on the preprocessed RAW data of the current Nth frame, the AE algorithm is used to calculate the scene parameter value of the Nth frame. The calculated scene parameter value is then used to determine whether the scene of the Nth frame is a low-brightness scene. Specifically, when acquiring the RAW data of the Nth frame, the mobile phone 100 can obtain the photosensitivity information of the corresponding Nth frame and the average brightness corresponding to the previous RAW data acquired by the sensor, and use the AE algorithm to calculate the current illuminance value.

[0128] For example, the ISO sensitivity information is extracted from the RAW data of the Nth frame of a video image, and the average brightness value of the previous RAW data (image) is obtained. Considering scene changes, ISO sensitivity information, and known camera parameters, an illuminance calculation model is established to analyze possible scene changes between the current and previous video frames, such as object movement or changes in light sources. The calculation based on the obtained average brightness is then appropriately adjusted and corrected. Assuming the established model is: Illuminance value = k * ISO sensitivity * Average brightness / Correction factor, where k is a constant determined based on camera characteristics, and the correction factor is used to account for scene changes. If the average brightness of the previous frame is 100, the ISO sensitivity is 800, k is 0.1, and the correction factor is 0.8, then the calculated illuminance value = 0.1 * 800 * 100 / 0.8 = 10000. It is understood that the above illuminance calculation method is only a simple example; more complex methods can be used to obtain more accurate results in actual calculations, but specific methods are not required here.

[0129] In other implementations, the mobile phone 100 can use the scene parameter values ​​obtained during the generation of the (N-1)th frame of the video frame. Specifically, during the generation of the (N-1)th frame of the video frame, regardless of whether the RAW data is processed using the AIRAW algorithm for noise reduction, the mobile phone 100 will subsequently convert the preprocessed RAW data of the (N-1)th frame of the video frame into YUV data to obtain the scene parameter values. For example, the mobile phone 100 can convert the preprocessed RAW data of the current (N-1)th frame of the video frame into YUV data and use the calculated luminance component Y of the YUV data as the luminance value of the (N-1)th frame of the video frame. Furthermore, the calculated luminance component Y of the YUV data can be combined with data such as the camera's aperture to convert the Y component value into a quantity closer to the actual illuminance value. Alternatively, the Y component value can be directly quantized and used to represent the magnitude of the illuminance value. Therefore, after acquiring the RAW data of the Nth frame of video, the mobile phone 100 preprocesses the RAW data of the Nth frame to obtain the preprocessed RAW data of the Nth frame. Then, it uses the scene parameter values ​​obtained during the N-1th frame of video to determine whether the scene of the Nth frame is a low-brightness scene. Understandably, the mobile phone 100 can reuse computing resources at this point, using the scene parameter values ​​obtained based on the YUV data corresponding to the N-1th frame of video to determine whether the scene of the Nth frame is a low-brightness scene. The specific process will be explained later. Figure 7 Detailed explanations will not be elaborated here.

[0130] S604 corresponds to a low-light scene in the Nth frame of the video. The AIRAW algorithm is used to denoise the RAW data of the acquired Nth frame to obtain the denoised RAW data. Understandably, because the phone does not need to use the AIRAW algorithm every time, it reduces the phone's power consumption and improves its performance.

[0131] In some embodiments, the preprocessed RAW data is denoised based on a trained AI model for denoising to obtain denoised RAW data.

[0132] S605 displays and stores the Nth frame of video based on the noise-reduced RAW data.

[0133] In some embodiments, the mobile phone 100 can perform anchor point processing, HDR fusion, tone mapping, and color space conversion on the noise-reduced RAW data to generate YUV data. The generated YUV data is then post-processed to generate video frames, enabling the display and storage of the Nth frame of video. For example, the YUV data can be used to generate the Nth frame of video required for display, as well as video frame data to be saved as a recording file. Post-processing of the generated YUV data can include the following operations: noise reduction and pixel processing. Noise reduction is used to reduce image noise, and the pixel processing system is used to process each pixel of the image. Specifically, noise reduction and pixel processing include hardware noise reduction, resizing, color correction, and detail enhancement.

[0134] In addition, it is understandable that the AIRAW algorithm is used to denoise RAW data, and the AIRAW algorithm has a certain time delay, which may cause a delay in the display of the first frame of video in some cases.

[0135] S606, for a scene corresponding to the Nth frame of video that is not a low-brightness scene, displays and stores the Nth frame of video based on the RAW data of the Nth frame of video.

[0136] In some embodiments, the mobile phone 100 performs anchor point processing, HDR fusion, tone mapping, and color space conversion on the preprocessed RAW data to generate YUV data. The generated YUV data is then processed to obtain the Nth frame of video, which is then displayed and stored.

[0137] Figure 7 According to an embodiment of this application, a schematic diagram of another video processing method is shown. This process is illustrated using the scene parameter values ​​obtained from the previous video frame as an example to determine whether the scene of the current video frame is a low-brightness scene. This allows for the determination of whether the preprocessed RAW data obtained from the current video frame needs to be processed using the AIRAW algorithm. The executing entity in this process can be any device with camera hardware and camera applications, such as the aforementioned mobile phone 100. The specific steps are as follows:

[0138] S701, acquire the RAW data T1 of the Nth frame of video footage captured by the image sensor for the current scene.

[0139] In some embodiments, the image sensor captures video of the scene corresponding to the current Nth frame of video, and obtains RAW data T1 of the Nth frame of video.

[0140] For example, the RAW data T1 of the Nth frame video may include the long frame RAW data G1 of the Nth frame video and the short frame RAW data F1 of the Nth frame video.

[0141] S702, perform RAW data preprocessing on the RAW data T1 of the Nth frame of the video to obtain the RAW data T2 of the Nth frame of the video.

[0142] Understandably, the specific preprocessing operations are described in S602 above and will not be repeated here.

[0143] For example, the RAW data T2 of the Nth frame video may include the long frame RAW data G2 of the Nth frame video and the short frame RAW data F2 of the Nth frame video.

[0144] S703, based on the scene parameter values ​​of the corresponding (N-1)th frame video image, determine whether noise reduction is needed for the RAW data T2 of the Nth frame video image. If it is determined that noise reduction is needed for the RAW data T2 of the Nth frame video image, proceed to S705; if it is determined that noise reduction is not needed for the RAW data T2 of the Nth frame video image, proceed to S704.

[0145] In some implementations, the mobile phone 100 can determine whether the scene of the Nth frame of video is a low-brightness scene based on whether the scene parameter value of the (N-1)th frame of video meets the scene parameter condition. If the scene parameter value of the corresponding N-1th frame of video meets the scene parameter condition, it indicates that the current scene is a low-brightness scene, and it is determined that noise reduction needs to be performed on the RAW data T2 of the Nth frame of video, proceeding to S705; if the scene parameter value of the N-1th frame of video does not meet the scene parameter condition, it indicates that the current scene is not a low-brightness scene, and it is determined that noise reduction does not need to be performed on the RAW data T2 of the Nth frame of video, proceeding to S704.

[0146] The scene parameter values ​​can be illuminance and brightness. If the illuminance value is less than the first illuminance threshold, the current scene is considered a low-brightness scene, and it is determined that noise reduction needs to be performed on the RAW data T2 of the Nth frame of the video, proceeding to S705; otherwise, the current scene is considered not a low-brightness scene, and it is determined that noise reduction does not need to be performed on the RAW data T2 of the Nth frame of the video, proceeding to S704.

[0147] If the brightness value is less than the first brightness threshold, the current scene is considered to be a low-brightness scene, and it is determined that the RAW data T2 of the Nth frame of the video needs to be denoised, and the process proceeds to S705; otherwise, the current scene is considered not to be a low-brightness scene, and it is determined that the RAW data T2 of the Nth frame of the video does not need to be denoised, and the process proceeds to S704.

[0148] S704, Based on the RAW data T2 of the Nth frame video frame, generate the YUV data K1 of the Nth frame video frame.

[0149] Understandably, this step is essentially the same as S404 above. The process of generating YUV data from RAW data can be referred to the description of S404 above, and will not be repeated here.

[0150] For example, the long frame RAW data G2 of the Nth frame video and the short frame RAW data F2 of the Nth frame video are fused to generate the YUV data K1 of the Nth frame video.

[0151] S705 uses the AIRAW algorithm to denoise the RAW data T2 of the Nth frame of the video, and obtains the RAW data T3 of the Nth frame of the video.

[0152] Understandably, this step is essentially the same as S604 above. The noise reduction process of the AIRAW algorithm can be referred to the description of S604 above, and will not be repeated here.

[0153] For example, the long frame RAW data G2 of the Nth frame of the video is denoised using the AIRAW algorithm to obtain long frame RAW data G3. The short frame RAW data F2 of the Nth frame of the video is not denoised using the AIRAW algorithm, that is, it is not processed by the AI ​​model, and the short frame RAW data F2 is taken as short frame RAW data F3.

[0154] S706, Based on the RAW data T3 of the Nth frame video image, generate the YUV data K1 of the Nth frame video image.

[0155] Understandably, this step is essentially the same as the process of generating YUV data from RAW data described in S605 above, and will not be repeated here.

[0156] For example, the long frame RAW data G3 of the Nth frame video and the short frame RAW data F3 of the Nth frame video are fused to generate the YUV data K1 of the Nth frame video.

[0157] S707, Perform YUV data processing on the YUV data K1 of the Nth frame video frame to obtain the YUV data K2 of the Nth frame video frame.

[0158] It is understandable that this step is essentially the same as the process of processing YUV data described in S605 above, and will not be repeated here.

[0159] S708, obtain scene parameter values ​​based on the YUV data K2 of the Nth frame of video.

[0160] In some embodiments, the mobile phone 100 can estimate the scene parameter values ​​of the Nth frame of video using a pre-configured scene parameter calculation model. For example, for the brightness value, the Y component can be extracted from the YUV data of the Nth frame of video, and the brightness value can be obtained based on the Y component.

[0161] For example, specialized image processing libraries or algorithms can be used to combine the YUV data of the Nth frame of video with other relevant information, such as camera characteristics and shooting environment, to estimate the illuminance as accurately as possible. For instance, the Y component is first extracted from the YUV data of the Nth frame of video. Then, combined with data such as the camera's aperture, the Y component value is converted into a value closer to the actual illuminance. Alternatively, the Y component value can be directly quantized to represent the magnitude of the illuminance. The specific calculation method is not required here.

[0162] Furthermore, it is understandable that the scene parameter values ​​obtained from the Nth frame of the video can be used as a condition for determining whether noise reduction is needed for the RAW data T2 of the (N+1)th frame of the video.

[0163] S709, based on the YUV data K2 of the Nth frame of the video, displays the video frame of the Nth frame.

[0164] Understandably, this step can refer to the description of displaying video frames in S605 and S606 above, and will not be repeated here.

[0165] Figure 8 According to an embodiment of this application, another video processing apparatus 50 is shown. Specifically, the video processing apparatus 50 includes: a sensor data acquisition module 50, a RAW data preprocessing module 51, an AIRAW algorithm module 52, and a generation module 53.

[0166] The sensor data acquisition module 50 is used to acquire the first RAW data of the Nth frame of video captured by the image sensor during the video acquisition process. In some implementations, the sensor data acquisition module 50 can be the aforementioned... Figure 5A Sensor node 300 (the first RAW data of the Nth frame of video can be...) Figure 5A The RAW data J1 in the image. In some other implementations, the sensor data acquisition module 50 can be the aforementioned... Figure 5B Sensor node 400 (the first RAW data of the Nth frame of video can be...) Figure 5B The RAW data (T1) in the data acquisition module 50 can be found in the above description. Figure 5A and Figure 5B The descriptions of sensor nodes 300 and 400 are omitted here.

[0167] RAW data preprocessing module 51 is used to preprocess the first RAW data of the Nth frame of video to obtain the second RAW data of the Nth frame of video. In some implementations, RAW data preprocessing module 51 can be the aforementioned Figure 5A The thin front-end node 301 (the second RAW data of the Nth frame of video can be) Figure 5A In some implementations, the RAW data preprocessing module 51 can be the aforementioned RAW data J2). Figure 5B The thin front-end node 401 (the second RAW data of the Nth frame of video can be...) Figure 5B The RAW data T2 is described above. For details on the functions of the RAW data preprocessing module 51, please refer to the above. Figure 5A and Figure 5B The descriptions of the thin front-end node 301 and thin front-end node 401 are not repeated here.

[0168] The AIRAW algorithm module 52 is used to perform noise reduction processing on the second RAW data of the Nth frame of video when the shooting scene corresponding to the Nth frame belongs to the first scene (e.g., a low-light scene) using an AI model to obtain the third RAW data of the Nth frame of video; when the shooting scene corresponding to the Nth frame of video does not belong to the first scene, no noise reduction processing is performed on the second RAW data of the Nth frame of video using an AI model. In some implementations, the AIRAW algorithm module 52 can be the above... Figure 5A The AIRAW algorithm node 303 (the third RAW data of the Nth frame video is RAW data J3). In some other implementations, the AIRAW algorithm module 52 can be the above-mentioned... Figure 5B The AIRAW algorithm node 403 (the third RAW data of the Nth frame video is RAW data T3). For the specific functions of the AIRAW algorithm module 52, please refer to the above. Figure 5A and Figure 5B The descriptions of AIRAW algorithm nodes 303 and AIRAW algorithm nodes 403 are not repeated here.

[0169] The generation module 53 is used to obtain the Nth frame video based on the third RAW data of the Nth frame video when the shooting scene corresponding to the Nth frame video belongs to the first scene, and to obtain the Nth frame video based on the second RAW data of the Nth frame video when the shooting scene corresponding to the Nth frame video does not belong to the first scene.

[0170] In some embodiments, the generation module includes: a YUV data generation module S1, a YUV data processing module S2, and a video frame generation module S3.

[0171] The YUV data generation module S1 is used to generate first YUV data for the Nth frame of video based on the third RAW data of the Nth frame when the shooting scene corresponding to the Nth frame belongs to the first scene, and to generate first YUV data for the Nth frame of video based on the second RAW data of the Nth frame when the shooting scene corresponding to the Nth frame does not belong to the first scene. In some implementations, the YUV data generation module S1 can be as described above. Figure 5A The offline front-end node 304 (the first YUV data is YUV data V1) is mentioned above. In other implementations, the YUV data generation module S1 can be the aforementioned... Figure 5B The offline front-end node 404 (the first YUV data is YUV data K1) is described above. For details on the function of the YUV data generation module S1, please refer to the above. Figure 5A and Figure 5B The descriptions of offline front-end node 304 and offline front-end node 404 are not repeated here.

[0172] The YUV data processing module S2 is used to post-process the first YUV data of the Nth frame video frame into second YUV data of the Nth frame video frame when the shooting scene corresponding to the Nth frame video frame belongs to the first scene, and to post-process the first YUV data of the Nth frame video frame into second YUV data of the Nth frame video frame when the shooting scene corresponding to the Nth frame video frame does not belong to the first scene; wherein, the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement. In some implementations, the YUV data processing module S2 can be one of the above. Figure 5A The image front-end engine node 305 (the second YUV data is YUV data V2) in the image front-end engine. In some other implementations, the YUV data processing module S2 can be the aforementioned... Figure 5B The image front-end engine node 405 (the second YUV data is YUV data K2) is described above. The specific functions of the YUV data processing module S2 can be found in the above description. Figure 5A and Figure 5B The descriptions of image front-end engine nodes 305 and 405 are omitted here.

[0173] The video frame generation module S3 is used to obtain the Nth video frame based on the second YUV data of the Nth video frame. In some implementations, the video frame generation module S3 can be as described above. Figure 5A The preview and recording processing node 306 in the [reference to a specific implementation]. In other implementations, the YUV data processing module S2 can be the aforementioned [specific implementation]. Figure 5B The preview and recording processing node 406 is described above. The specific functions of the video frame generation module S3 can be found in the above description. Figure 5A and Figure 5B The descriptions of the preview and recording processing nodes 306 and 406 are omitted here.

[0174] Figure 9 According to an embodiment of this application, a schematic diagram of the hardware structure of a mobile phone 100 is shown.

[0175] Mobile phone 100 is capable of executing the video processing method provided in the embodiments of this application. Figure 9 In this context, similar components share the same reference numerals. For example... Figure 9 As shown, the mobile phone 100 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0176] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0177] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. Processor 110 can execute the video processing methods provided in the embodiments of this application. For example, executing the above... Figure 6 , Figure 7 The flowchart of the video processing method is shown.

[0178] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a Micro LED, a Micro OLED, a quantum dot light-emitting diode (QLED), etc.

[0179] Camera 170 is used to capture still images or videos. An optical image of the scene, generated by the lens, is projected onto a photosensitive element (such as an image sensor). The photosensitive element converts the light signal into an electrical signal, and under specific settings and conditions, obtains RAW data. Mobile phone 100 can implement the functions corresponding to the video processing methods in the above embodiments through the cooperation of an ISP, video codec, GPU, display screen, and application processor. For example, the image signal processor preprocesses the RAW data acquired by the sensor; the graphics processor can determine whether the scene parameters meet the scene parameter conditions; if the scene parameter conditions are met, the AIRAW algorithm is used for noise reduction to obtain denoised RAW data. The denoised RAW data is converted into YUV data by the image signal processor, resulting in video frame data that can be previewed and used to generate video recording files. The CPU displays the video frames and generates and saves the video recording file.

[0180] The sensor module 190 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc. In some embodiments, the ambient light sensor or the proximity sensor may be used to detect the brightness of the environment during video recording, to assist in calculating the scene parameter values ​​of the Nth frame of video in the video processing methods of the above embodiments.

[0181] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0182] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0183] This application also provides a computer program product that, when executed on an electronic device, enables the electronic device to implement the video processing methods provided in the foregoing embodiments.

[0184] This application also provides a computer-readable storage medium storing one or more programs, which, when executed by an electronic device, enable the electronic device to implement the video processing methods provided in the foregoing embodiments.

[0185] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0186] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0187] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0188] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored thereon on one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc-read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0189] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0190] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0191] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0192] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A video processing method applied to electronic devices, characterized in that, The method includes: During the video acquisition process, the first RAW data of the Nth frame of video captured by the image sensor of the electronic device is obtained; The first RAW data of the Nth frame of video is preprocessed to obtain the second RAW data of the Nth frame of video; If the shooting scene corresponding to the Nth frame of video belongs to the first scene, the AI ​​model is used to perform noise reduction processing on the second RAW data of the Nth frame of video to obtain the third RAW data of the Nth frame of video, and the Nth frame of video is obtained based on the third RAW data of the Nth frame of video. If the shooting scene corresponding to the Nth frame of video does not belong to the first scene, no AI model is used to perform noise reduction processing on the second RAW data of the Nth frame of video, and the Nth frame of video is obtained based on the second RAW data of the Nth frame of video.

2. The method according to claim 1, characterized in that, The first scene is a scene whose scene parameters satisfy the first condition, and the scene parameters include brightness value and / or illuminance value.

3. The method according to claim 2, characterized in that, The first condition includes: the brightness value is less than a first brightness threshold, and / or the illuminance value is less than a first illuminance threshold.

4. The method according to claim 2 or 3, characterized in that, The shooting scene corresponding to the Nth frame of video belongs to the first scene, including: the scene parameters of the Nth frame of video satisfy the first condition; and the scene parameters of the Nth frame of video are obtained based on the sensitivity information corresponding to the first RAW data of the Nth frame of video.

5. The method according to claim 2 or 3, characterized in that, The shooting scene corresponding to the Nth video frame belongs to the first scene, including: the scene parameters of the Nth video frame satisfy the first condition; and... The shooting scene corresponding to the (N-1)th video frame belongs to the first scene, and the scene parameters of the Nth video frame are obtained based on the YUV data corresponding to the third RAW data of the (N-1)th video frame. The shooting scene corresponding to the (N-1)th video frame does not belong to the first scene, and the scene parameters of the Nth video frame are obtained based on the YUV data corresponding to the second RAW data of the (N-1)th video frame.

6. The method according to claim 5, characterized in that, The scene parameters of the Nth frame of video are obtained based on the YUV data corresponding to the third RAW data of the (N-1)th frame of video, including: The scene parameters of the Nth frame video are obtained based on the third RAW data of the (N-1)th frame video in the following manner: The third RAW data of the N-1th video frame is converted into the first YUV data of the N-1th video frame; The first YUV data of the N-1th frame video is post-processed into the second YUV data of the N-1th frame video, wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement. The luminance component is calculated based on the second YUV data of the (N-1)th frame of the video, and the scene parameters of the Nth frame of the video are obtained based on the luminance component.

7. The method according to claim 5, characterized in that, The scene parameters of the Nth frame of video are obtained based on the YUV data corresponding to the second RAW data of the (N-1)th frame of video, including: The scene parameters of the Nth frame of video are obtained based on the second RAW data of the (N-1)th frame of video in the following manner: Convert the second RAW data of the N-1th frame of video into the first YUV data of the N-1th frame of video; The first YUV data of the N-1th frame video is post-processed into the second YUV data of the N-1th frame video, wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement. The luminance component is calculated based on the second YUV data of the (N-1)th frame of the video, and the scene parameters of the Nth frame of the video are obtained based on the luminance component.

8. The method according to claim 1, characterized in that, The second RAW data includes a first long frame data and a first short frame data, and the third RAW data includes a second long frame data and a second short frame data; Furthermore, when the shooting scene corresponding to the Nth frame of the video belongs to the first scene, the third RAW data is obtained by using an AI model to perform noise reduction processing on the second RAW data, including: The first long frame data is denoised using an AI model to obtain the second long frame data; the first short frame data is directly used as the second short frame data.

9. The method according to claim 1, characterized in that, The video acquisition process includes the process of shooting video through a first mode, wherein the first mode is either night scene mode or high dynamic range mode.

10. The method according to claim 1, characterized in that, The Nth video frame is obtained based on the third RAW data of the Nth video frame, including: The third RAW data of the Nth frame of video is converted into the first YUV data of the Nth frame of video; The first YUV data of the Nth frame video is post-processed into the second YUV data of the Nth frame video, wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement. The Nth frame of video is obtained based on the second YUV data of the Nth frame.

11. A video processing apparatus, characterized in that, include: The sensor data acquisition module is used to acquire the first RAW data of the Nth frame of video captured by the image sensor during the video acquisition process. The RAW data preprocessing module is used to preprocess the first RAW data of the Nth frame video to obtain the second RAW data of the Nth frame video. The AIRAW algorithm module is used to perform noise reduction processing on the second RAW data of the Nth frame video image when the shooting scene corresponding to the Nth frame video image belongs to the first scene, so as to obtain the third RAW data of the Nth frame video image; when the shooting scene corresponding to the Nth frame video image does not belong to the first scene, the AI ​​model is not used to perform noise reduction processing on the second RAW data of the Nth frame video image. The generation module is used to obtain the Nth video frame based on the third RAW data of the Nth video frame when the shooting scene corresponding to the Nth video frame belongs to the first scene. And when the shooting scene corresponding to the Nth frame of video does not belong to the first scene, the Nth frame of video is obtained based on the second RAW data of the Nth frame of video.

12. The apparatus according to claim 11, characterized in that, The generation module includes: The YUV data generation module is used to generate first YUV data of the Nth frame video based on the third RAW data of the Nth frame video when the shooting scene corresponding to the Nth frame video belongs to the first scene, and to generate first YUV data of the Nth frame video based on the second RAW data of the Nth frame video when the shooting scene corresponding to the Nth frame video does not belong to the first scene. The YUV data processing module is configured to, when the shooting scene corresponding to the Nth frame of video belongs to a first scene, post-process the first YUV data of the Nth frame of video into the second YUV data of the Nth frame of video; and to, when the shooting scene corresponding to the Nth frame of video does not belong to the first scene, post-process the first YUV data of the Nth frame of video into the second YUV data of the Nth frame of video; wherein the post-processing includes at least one of the following: hardware noise reduction, multi-frame super-resolution, resizing, color correction, and detail enhancement. The video frame generation module is used to obtain the Nth frame video based on the second YUV data of the Nth frame video.

13. An electronic device, characterized in that, include: One or more processors; One or more memories; the one or more memories storing one or more programs, which, when executed by the one or more processors, cause the electronic device to perform the video processing method of any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on a computer, cause the computer to perform the video processing method according to any one of claims 1 to 10.

15. A computer program product, the computer program product comprising: Computer program code that, when run on a computer, causes the computer to perform the video processing method as described in any one of claims 1 to 10.

Images