Video enhancement method, model training method and device
The model is determined through pre-trained enhancement parameters, and the offset matrix and weight matrix are generated, which solves the registration problem in RAW image video enhancement, achieves high-quality video enhancement effects, and improves the clarity and detail fidelity of the moving video.
Patent Information
- Application Number
- CN202310650683.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-06-02
AI Technical Summary
The prior art is difficult to effectively enhance the video of moving video, especially when the video frame is a RAW image, accurate registration cannot be carried out, resulting in low video quality after enhancement.
The model is determined by pre-trained enhancement parameters, and the feature extraction network, the offset prediction network and the weight prediction network are generated, and the offset matrix and the weight matrix are used to determine the mapping relationship between video frames, thereby performing video enhancement.
It achieves high-quality enhancement of RAW image video, improves mapping accuracy and enhancement effect between video frames, and improves the clarity and detail fidelity of the video.
Smart Images

Figure CN119090755A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a video enhancement method, a model training method and a device. Background Art
[0002] When the acquisition device acquires video, it will be affected by the acquisition environment. For example, when the acquisition environment is a low-light environment at night, due to the limited number of incident photons on the sensor of the acquisition device, the signal-to-noise ratio of each video frame in the acquired video may be low, and the clarity of the video is also low. Video enhancement uses the redundant information of each video frame in the time domain and the correlation between each video frame in the video to process each video frame in the video to improve the signal-to-noise ratio of each video frame in the video, thereby improving the clarity of the video, that is, improving the sensory quality of the video, and accordingly, can improve the user's experience of watching the video. For the video in the static scene acquired by the static acquisition device (which can be called static video), the 3D DNR (3D Digital Noise Reducer) technology can be used to enhance the static video to obtain a high-quality enhanced video. However, when the acquisition device is moving, or the video in the motion scene is captured by the acquisition device (which can be called motion video), if the 3D DNR technology is used to enhance the motion video, the quality of the enhanced video is low, and the video frames in the enhanced video often have serious noise, blurred details, color distortion, motion ghosting and other problems. In order to achieve video enhancement of motion video, for each video frame in the motion video, it is necessary to use multiple video frames in the motion video to enhance the video frame to enhance the information of the motion area in the video frame.
[0003] In the prior art, when enhancing a video, the video frames in the video can be firstly registered by a preset registration method (for example, a registration method based on feature point matching, or a registration method based on block matching technology). Registration is to determine the mapping relationship between video frames, such as determining the feature points corresponding to feature points in one video frame in another video frame, or determining the image area corresponding to an image area in one video frame in another video frame. Subsequently, the video enhancement is performed according to the mapping relationship between the video frames.
[0004] However, the preset registration method is applicable to RGB (Red, Green, Blue) images. The pixel value of RGB images has a nonlinear relationship with the intensity of incident light, and it is difficult to analyze RGB images. The pixel value of RAW (original) images has a good linear relationship with the intensity of incident light, which retains the most original information of the image. Therefore, based on the physical imaging process, it is less difficult to accurately model the pixel value of RAW images, that is, it is less difficult to analyze RAW images. However, the preset registration method may not be applicable to RAW images. When each video frame in the video is a RAW image, it is impossible to register each video frame in the video, and it is also impossible to enhance the video. Moreover, the accuracy of registering video frames in the video using the preset registration method is low. For example, when registering video frames using a registration method based on feature point matching, since the feature points in each video frame are not evenly distributed in the video frame, it is impossible to determine the corresponding feature points in another video frame for an area in which no feature points are detected in one video frame; or, when registering video frames using a registration method based on block matching technology, multiple image areas in one video frame may correspond to the same image area in another video frame, and the enhanced video frame may have ghosting, which results in low quality of the enhanced video. Summary of the invention
[0005] The purpose of the embodiments of the present invention is to provide a video enhancement method, a model training method and a device to enhance the RAW image and improve the quality of the obtained target video. The specific technical solution is as follows:
[0006] In a first aspect of an embodiment of the present invention, a video enhancement method is first provided, the method comprising:
[0007] Determine a designated reference video frame from each video frame of the original video; wherein each video frame in the original video is a RAW image;
[0008] Inputting the reference video frame and adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model;
[0009] The elements in the offset matrix represent the offset of the first image region in the reference video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image region in the reference video frame and the second image region in the adjacent video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region; if an element corresponds to the second image region in the adjacent video frame, the element represents the weight of the corresponding second image region;
[0010] For each first image region in the reference video frame, based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame, determining a second image region corresponding to the first image region in the adjacent video frame;
[0011] For each first image region in the reference video frame, according to the weight of the first image region and the weight of the second image region corresponding to the first image region in the adjacent video frame, the enhanced pixel value of the first image region is calculated to obtain a target video frame corresponding to the reference video frame;
[0012] Based on the target video frame, a target video corresponding to the original video is obtained.
[0013] Optionally, the enhanced parameter determination model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network;
[0014] The step of inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model includes:
[0015] Inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model, performing feature extraction on the reference video frame and the adjacent video frame through the first feature extraction network, and obtaining a feature vector representing the image features of the reference video frame and the image features of the adjacent video frame as a first feature vector;
[0016] Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector;
[0017] Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix;
[0018] Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector;
[0019] The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
[0020] Optionally, the determining, for each first image region in the reference video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame, includes:
[0021] For each first image region in the reference video frame, obtaining the coordinates of the first image region in the reference video frame;
[0022] Calculating a sum of the coordinates of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame as a reference coordinate;
[0023] In the adjacent video frames, an image region corresponding to the reference coordinates is determined to obtain a second image region corresponding to the first image region in the adjacent video frames.
[0024] Optionally, for each first image region in the reference video frame, calculating the enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame to obtain a target video frame corresponding to the reference video frame includes:
[0025] For each pixel point in the first image area in the reference video frame, if the reference coordinate corresponding to the pixel point in the adjacent video frame is an integer, according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, a weighted sum of the pixel value of the pixel point and the pixel value of the pixel point at the reference coordinate is calculated to obtain a target video frame corresponding to the reference video frame;
[0026] If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, the pixel value corresponding to the reference coordinate is determined based on the pixel values of the pixel points adjacent to the reference coordinate, and the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate is calculated according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
[0027] Optionally, determining the pixel value corresponding to the reference coordinate based on the pixel values of the pixel points adjacent to the reference coordinate includes:
[0028] Based on a nearest neighbor interpolation algorithm, determine the pixel value of the pixel point with the smallest distance from the reference coordinate as the pixel value corresponding to the reference coordinate;
[0029] or,
[0030] Based on a bilinear interpolation algorithm, the pixel value corresponding to the reference coordinate is calculated based on the pixel values of a preset number of pixel points within a preset neighborhood range of the reference coordinate.
[0031] Optionally, the adjacent video frames of the reference video frame include: a video frame located before the reference video frame in the original video, and / or a video frame located after the reference video frame.
[0032] In a second aspect of an embodiment of the present invention, a model training method is provided for generating an enhanced parameter determination model as described in any one of the first aspects above, the method comprising:
[0033] Acquire a first sample video frame and a second sample video frame; wherein the first sample video frame and the second sample video frame are RAW images; and the first sample video frame is obtained by performing noise addition processing on the second sample video frame;
[0034] Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, and obtaining the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure;
[0035] The elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the first sample video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area;
[0036] For each first image region in the first sample video frame, based on a position of the first image region in the first sample video frame and an offset of the first image region relative to the adjacent video frame, determining a second image region corresponding to the first image region in the adjacent video frame;
[0037] For each first image region in the first sample video frame, according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, an enhanced pixel value of the first image region is calculated to obtain a predicted video frame corresponding to the first sample video frame;
[0038] calculating a loss function value representing a difference between the predicted video frame and the second sample video frame;
[0039] Based on the calculated loss function value, the model parameters of the enhancement parameter determination model of the initial structure are adjusted until a preset convergence condition is reached to obtain a trained enhancement parameter determination model.
[0040] Optionally, the enhancement parameter determination model of the initial structure includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network;
[0041] The step of inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure to obtain the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure includes:
[0042] Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, performing feature extraction on the first sample video frame and the adjacent video frames through the first feature extraction network, and obtaining a feature vector representing the image features of the first sample video frame and the image features of the adjacent video frames as a first feature vector;
[0043] Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector;
[0044] Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix;
[0045] Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector;
[0046] The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
[0047] In a third aspect of the embodiments of the present invention, a video enhancement device is provided, the device comprising:
[0048] A reference video frame determination module, used to determine a designated reference video frame from each video frame of an original video; wherein each video frame in the original video is a RAW image;
[0049] A matrix acquisition module, used for inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model;
[0050] The elements in the offset matrix represent the offset of the first image region in the reference video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image region in the reference video frame and the second image region in the adjacent video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region; if an element corresponds to the second image region in the adjacent video frame, the element represents the weight of the corresponding second image region;
[0051] an image region determination module, configured to determine, for each first image region in the reference video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame;
[0052] a target video frame acquisition module, configured to calculate, for each first image region in the reference video frame, an enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, so as to obtain a target video frame corresponding to the reference video frame;
[0053] The target video acquisition module is used to obtain the target video corresponding to the original video based on the target video frame.
[0054] Optionally, the enhanced parameter determination model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network;
[0055] The matrix acquisition module is specifically used for:
[0056] Inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model, performing feature extraction on the reference video frame and the adjacent video frame through the first feature extraction network, and obtaining a feature vector representing the image features of the reference video frame and the image features of the adjacent video frame as a first feature vector;
[0057] Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector;
[0058] Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix;
[0059] Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector;
[0060] The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
[0061] Optionally, the image area determination module is specifically used to:
[0062] For each first image region in the reference video frame, obtaining the coordinates of the first image region in the reference video frame;
[0063] Calculating a sum of the coordinates of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame as a reference coordinate;
[0064] In the adjacent video frames, an image region corresponding to the reference coordinates is determined to obtain a second image region corresponding to the first image region in the adjacent video frames.
[0065] Optionally, the target video frame acquisition module is specifically used to:
[0066] For each pixel point in the first image area in the reference video frame, if the reference coordinate corresponding to the pixel point in the adjacent video frame is an integer, according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, a weighted sum of the pixel value of the pixel point and the pixel value of the pixel point at the reference coordinate is calculated to obtain a target video frame corresponding to the reference video frame;
[0067] If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, the pixel value corresponding to the reference coordinate is determined based on the pixel values of the pixel points adjacent to the reference coordinate, and the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate is calculated according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
[0068] Optionally, the target video frame acquisition module is specifically used to:
[0069] Based on a nearest neighbor interpolation algorithm, determine the pixel value of the pixel point with the smallest distance from the reference coordinate as the pixel value corresponding to the reference coordinate;
[0070] or,
[0071] Based on a bilinear interpolation algorithm, the pixel value corresponding to the reference coordinate is calculated based on the pixel values of a preset number of pixel points within a preset neighborhood range of the reference coordinate.
[0072] Optionally, the adjacent video frames of the reference video frame include: a video frame located before the reference video frame in the original video, and / or a video frame located after the reference video frame.
[0073] In a fourth aspect of the embodiments of the present invention, a model training device is provided for generating the enhancement parameter determination model described in any one of the first aspects above, the device comprising:
[0074] A sample video frame acquisition module, used to acquire a first sample video frame and a second sample video frame; wherein the first sample video frame and the second sample video frame are RAW images; and the first sample video frame is obtained by performing noise addition processing on the second sample video frame;
[0075] A matrix acquisition module, used for inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, and obtaining the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure;
[0076] The elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the first sample video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area;
[0077] an image region determining module, configured to determine, for each first image region in the first sample video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the first sample video frame and an offset of the first image region relative to the adjacent video frame;
[0078] a predicted video frame acquisition module, configured to calculate, for each first image region in the first sample video frame, an enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, so as to obtain a predicted video frame corresponding to the first sample video frame;
[0079] a difference calculation module, configured to calculate a loss function value representing a difference between the predicted video frame and the second sample video frame;
[0080] The adjustment module is used to adjust the model parameters of the enhancement parameter determination model of the initial structure based on the calculated loss function value until a preset convergence condition is reached to obtain a trained enhancement parameter determination model.
[0081] Optionally, the enhancement parameter determination model of the initial structure includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network;
[0082] The matrix acquisition module is specifically used for:
[0083] Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, performing feature extraction on the first sample video frame and the adjacent video frames through the first feature extraction network, and obtaining a feature vector representing the image features of the first sample video frame and the image features of the adjacent video frames as a first feature vector;
[0084] Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector;
[0085] Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix;
[0086] Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector;
[0087] The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
[0088] In a fifth aspect of an embodiment of the present invention, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0089] Memory, used to store computer programs;
[0090] The processor is used to implement any video enhancement method described in the first aspect when executing the program stored in the memory.
[0091] In a sixth aspect of the embodiments of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0092] Memory, used to store computer programs;
[0093] The processor is used to implement the model training method described in any one of the second aspects above when executing the program stored in the memory.
[0094] In a seventh aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the video enhancement method described in any one of the first aspects is implemented.
[0095] In an eighth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the model training method described in any one of the second aspects is implemented.
[0096] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the steps of the video enhancement method described in the first aspect.
[0097] An embodiment of the present invention also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the steps of the model training method described in the second aspect above.
[0098] A video enhancement method provided by an embodiment of the present invention determines a designated reference video frame from each video frame of an original video; wherein each video frame in the original video is a RAW image; the reference video frame and the adjacent video frames of the reference video frame are input into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model; wherein the elements in the offset matrix represent the offset of a first image region in the reference video frame relative to an adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image region in the reference video frame and the second image region in the adjacent video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region ; If an element corresponds to a second image area in an adjacent video frame, the element represents the weight of the corresponding second image area; for each first image area in a reference video frame, based on the position of the first image area in the reference video frame and the offset of the first image area relative to the adjacent video frame, determine the second image area corresponding to the first image area in the adjacent video frame; for each first image area in the reference video frame, according to the weight of the first image area and the weight of the second image area corresponding to the first image area in the adjacent video frame, calculate the enhanced pixel value of the first image area to obtain a target video frame corresponding to the reference video frame; based on the target video frame, obtain a target video corresponding to the original video.
[0099] Based on the above processing, the original video including each video frame as a RAW image can be enhanced based on the offset matrix and the weight matrix to obtain the target video corresponding to the original video, that is, the RAW image can be enhanced; and, since the offset matrix and the weight matrix are obtained based on the pre-trained enhancement parameter determination model, for each first image area in the reference video frame, the second image area corresponding to the first image area in the adjacent video frame of the reference video frame is determined based on the offset matrix, which can improve the accuracy of the determined second image area. Furthermore, based on the weights of each first image area in the reference video frame and the weights of the second image area corresponding to each first image area, the pair of reference video frames is enhanced to improve the quality of the obtained target video.
[0100] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0101] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0102] Figure 1 A first flow chart of a video enhancement method provided by an embodiment of the present invention;
[0103] Figure 2 A second flow chart of the video enhancement method provided by an embodiment of the present invention;
[0104] Figure 3 A third flow chart of the video enhancement method provided by an embodiment of the present invention;
[0105] Figure 4 A fourth flow chart of the video enhancement method provided by an embodiment of the present invention;
[0106] Figure 5 A fifth flow chart of the video enhancement method provided by an embodiment of the present invention;
[0107] Figure 6 A first flow chart of a model training method provided by an embodiment of the present invention;
[0108] Figure 7 A second flow chart of the model training method provided by an embodiment of the present invention;
[0109] Figure 8 A working principle diagram of an enhanced parameter determination model provided by an embodiment of the present invention;
[0110] Fig. 9 A comparative example diagram of the video enhancement effect provided by an embodiment of the present invention;
[0111] Fig.10 A structural diagram of a video enhancement device provided by an embodiment of the present invention;
[0112] Fig.11 A structural diagram of a model training device provided by an embodiment of the present invention;
[0113] Fig.12 A first structural diagram of an electronic device provided by an embodiment of the present invention;
[0114] Fig.13 A second structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0115] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.
[0116] In the related art, when performing video enhancement on a video, it is necessary to first determine the mapping relationship between video frames through a preset registration method, and then perform video enhancement based on the mapping relationship between video frames. However, the preset registration method is not applicable to RAW images. Therefore, when each video frame in the video is a RAW image, it is impossible to perform registration on each video frame in the video, and it is also impossible to perform video enhancement on the video; and the accuracy of registering the video frames in the video using the preset registration method is low, which results in low quality of the enhanced video.
[0117] In order to solve the above problems, an embodiment of the present invention provides a video enhancement method, which is applied to an electronic device. The electronic device can obtain an original video to be enhanced, and determine a reference video frame specified in the original video, and each video frame in the original video is a RAW image. Furthermore, the electronic device can determine the model through the pre-trained enhancement parameters according to the video enhancement method provided by the embodiment of the present invention to obtain an offset matrix and a weight matrix. For each first image area in the reference video frame, based on the position of the first image area in the reference video frame and the offset of the first image area relative to the adjacent video frame, the second image area corresponding to the first image area in the adjacent video frame is determined; for each first image area in the reference video frame, according to the weight of the first image area and the weight of the second image area corresponding to the first image area in the adjacent video frame, the enhanced pixel value of the first image area is calculated to obtain a target video frame corresponding to the reference video frame; based on the target video frame, the target video corresponding to the original video is obtained, so that the RAW image can be enhanced; and the quality of the obtained target video can be improved.
[0118] See also Figure 1 , Figure 1 A first flow chart of a video enhancement method provided by an embodiment of the present invention, the method may include the following steps:
[0119] S101: Determine a designated reference video frame from each video frame of an original video.
[0120] Each video frame in the original video is a RAW image.
[0121] S102: Input the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model.
[0122] Among them, the elements in the offset matrix represent the offset of the first image area in the reference video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the reference video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the reference video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area.
[0123] S103: For each first image region in the reference video frame, determine a second image region corresponding to the first image region in the adjacent video frame based on the position of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame.
[0124] S104: For each first image area in the reference video frame, based on the weight of the first image area and the weight of the second image area corresponding to the first image area in the adjacent video frame, calculate the enhanced pixel value of the first image area to obtain the target video frame corresponding to the reference video frame.
[0125] S105: Based on the target video frame, obtain a target video corresponding to the original video.
[0126] Based on the video enhancement method provided by the embodiment of the present invention, the original video including each video frame as a RAW image can be enhanced based on the offset matrix and the weight matrix to obtain the target video corresponding to the original video, that is, the RAW image can be enhanced; and, since the offset matrix and the weight matrix are obtained based on the pre-trained enhancement parameter determination model, for each first image area in the reference video frame, the second image area corresponding to the first image area in the adjacent video frame of the reference video frame is determined based on the offset matrix, which can improve the accuracy of the determined second image area. Furthermore, based on the weights of each first image area in the reference video frame and the weights of the second image area corresponding to each first image area, the pair of reference video frames is enhanced to improve the quality of the obtained target video.
[0127] For step S101, the original video is a video in RAW format without ISP (Image Signal Processing), each video frame in the original video is a RAW image, and the pixel value of each pixel in each RAW image is RAW domain data. RAW domain data is: the original data after the light signal recorded by the photosensitive element of the acquisition device is converted into a digital signal. The pixel value of a pixel point may include single-channel RAW domain data, for example, the pixel value of a pixel point may be the G (Green) value recorded by a photosensitive element.
[0128] The electronic device can determine the video frame that needs to be enhanced from each video frame of the original video to obtain a designated reference video frame. For example, each video frame in the original video can be a reference video frame, or some video frames in the original video can be reference video frames.
[0129] With respect to step S102 , the adjacent video frames of the reference video frame include: a video frame located before the reference video frame in the original video, and / or a video frame located after the reference video frame.
[0130] The adjacent video frames of the reference video frame include the video frames in the original video that are located before the reference video frame. When the reference video frame is the first video frame in the original video, since there is no video frame located before the reference video frame in the original video, the video frame in the original video that is located after the reference video frame can be selected as the adjacent video frame of the reference video frame.
[0131] The adjacent video frames of the reference video frame include the video frames in the original video that are located after the reference video frame. When the reference video frame is the last video frame in the original video, since there is no video frame located after the reference video frame in the original video, the video frame in the original video that is located before the reference video frame can be selected as the adjacent video frame of the reference video frame.
[0132] The adjacent video frames of the reference video frame include: a first number of video frames in the original video before the reference video frame, and a second number of video frames after the reference video frame. The first number and the second number can be set based on demand. When the quality of the target video needs to be improved, the first number and the second number can be set to a larger value; when the efficiency of obtaining the target video needs to be improved, the first number and the second number can be set to a smaller value. In addition, the first number and the second number can be the same, such as the first number and the second number can be 2. Alternatively, the first number and the second number can be different, such as the first number can be 1 and the second number can be 3.
[0133] The enhanced parameter determination model may be a CNN (Convolutional Neural Network) model. The enhanced parameter determination model is obtained by training based on the first sample video frame and the second sample video frame, and the first sample video frame is obtained by performing noise processing on the second sample video frame. The training method of the enhanced parameter determination model refers to the relevant introduction of the subsequent embodiments.
[0134] After acquiring the reference video frame, the electronic device can obtain a pixel matrix composed of pixel values of each pixel point of the reference video frame (which can be called a first pixel matrix). After acquiring the adjacent video frame of the reference video frame, the electronic device can obtain a pixel matrix of pixel values of each pixel point of the adjacent video frame (which can be called a second pixel matrix). Then, the electronic device can concatenate the first pixel matrix and the second pixel matrix to obtain a concatenated pixel matrix (which can be called a third pixel matrix).
[0135] Subsequently, the electronic device inputs the third pixel matrix into the pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model. The elements in the offset matrix represent the offset of the first image region in the reference video frame relative to the adjacent video frame of the reference video frame; the elements in the weight matrix correspond one-to-one with the first image region in the reference video frame and the second image region in the adjacent video frame of the reference video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region; if an element corresponds to the second image region in the adjacent video frame of the reference video frame, the element represents the weight of the corresponding second image region.
[0136] Exemplarily, the height and width dimensions of the third pixel matrix are h*w. The height and width dimensions of the offset matrix output by the enhanced parameter determination model are . An element in the offset matrix corresponds to a first image area in the reference video frame, and the size of a first image area in the reference video frame is 4*4. The element corresponding to the first image area in the offset matrix represents the offset of each pixel in the first image area relative to the adjacent video frame of the reference video frame. The offset of a first image area relative to the adjacent video frame of the reference video frame includes: the offset of the first image area in the horizontal direction relative to the adjacent video frame of the reference video frame, and the offset of the first image area in the vertical direction relative to the adjacent video frame of the reference video frame.
[0137] The enhancement parameters determine the height and width of the weight matrix output by the model: If an element in the weight matrix corresponds to a first image region in the reference video frame, the size of the first image region in the reference video frame is 4*4, and the element corresponding to the first image region in the weight matrix represents the weight of the first image region; if an element in the weight matrix corresponds to a second image region in an adjacent video frame of the reference video frame, the size of the second image region in the adjacent video frame of the reference video frame is 4*4, and the element corresponding to the second image region in the weight matrix represents the weight of the second image region.
[0138] In some embodiments, the enhanced parameter determination model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network, and a weight prediction network.
[0139] Accordingly, in Figure 1 Based on Figure 2 , step S102 comprises the following steps:
[0140] S1021: Input the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model, perform feature extraction on the reference video frame and the adjacent video frame through a first feature extraction network, and obtain a feature vector representing the image features of the reference video frame and the image features of the adjacent video frame as a first feature vector.
[0141] S1022: Perform feature extraction on the first feature vector through a second feature extraction network to obtain a second feature vector.
[0142] S1023: Normalize the second eigenvector through the offset prediction network to obtain a corresponding offset matrix.
[0143] S1024: Perform feature extraction on the first feature vector through a third feature extraction network to obtain a third feature vector.
[0144] S1025: Normalize the third eigenvector through the weight prediction network to obtain a corresponding weight matrix.
[0145] The first feature extraction network may include multiple convolutional layers, including: a convolutional layer with a convolution kernel of 3×3 and a stride of 1, and a convolutional layer with a convolution kernel of 3×3 and a stride of 2. In addition, each convolutional layer included in the first feature extraction network is also connected to an activation function, such as a ReLu (Rectified Linear Units) function.
[0146] The third pixel matrix is subjected to feature extraction through the multiple convolutional layers included in the first feature extraction network, and the third pixel matrix is downsampled to obtain a feature vector (i.e., a first feature vector) representing the image features of the reference video frame and the image features of the adjacent video frame of the reference video frame. If the height and width dimensions of the third pixel matrix are h*w, the electronic device inputs the third pixel matrix into the pre-trained enhancement parameter determination model, and then the third pixel matrix is subjected to feature extraction through the first feature network, and the height and width dimensions are obtained. The feature extraction result is used as the first feature vector. And an element of the first feature vector may correspond to a 4*4 region in the reference video frame or in a video frame adjacent to the reference video frame.
[0147] The second feature extraction network may include two convolutional layers, which are convolutional layers with a convolution kernel of 3×3 and a step size of 1, and the first convolutional layer in the second feature extraction network is connected to an activation function, such as the activation function may be a ReLu function; since the output data of the second feature extraction network is: the offset of each first image region in the reference video frame relative to the adjacent video frame of the reference video frame, and the offset includes the offset in the horizontal direction and the offset in the vertical direction, that is, each first image region in the reference video frame has two offsets relative to each adjacent video frame of the reference video frame, then each adjacent video frame of the reference video frame corresponds to two output channels of the second feature extraction network. Therefore, the number of output channels of the second feature extraction network is twice the number of adjacent video frames of the reference video frame. If the reference video frame corresponds to 4 adjacent video frames, the number of output channels of the second feature extraction network is 8.
[0148] The first feature vector is feature extracted by two convolutional layers included in the second feature extraction network to obtain a feature vector (i.e., a second feature vector) representing the offset of the reference video frame relative to the adjacent video frame of the reference video frame. Subsequently, the second feature vector is normalized by the offset prediction network to obtain a corresponding offset matrix. For example, the offset prediction network can use a normalization function to normalize the second feature vector, such as the normalization function can be a tanh (hyperbolic tangent) function, and the second feature vector is normalized to the interval [-1, 1] to obtain a corresponding offset matrix.
[0149] The third feature extraction network may include two convolutional layers, which are convolutional layers with a convolution kernel of 3×3 and a step size of 1, and the first convolutional layer in the third feature extraction network is connected with an activation function, such as the activation function may be a ReLu function; since the output data of the third feature extraction network are: the weights of each first image region in the reference video frame, and the weights of each second image region in the adjacent video frame of the reference video frame, that is, the reference video frame corresponds to an output channel of the third feature extraction network, and each adjacent video frame of the reference video frame also corresponds to an output channel of the third feature extraction network. Therefore, the number of output channels of the third feature extraction network is the same as the sum of the number of reference video frames and the number of adjacent video frames of the reference video frame. If a reference video frame corresponds to 4 adjacent video frames, the number of output channels of the third feature extraction network is 5.
[0150] The first feature vector is feature extracted by two convolutional layers included in the third feature extraction network to obtain a feature vector (i.e., a third feature vector) representing the weights of the reference video frame and the adjacent video frames of the reference video frame. Subsequently, the third feature vector is normalized by the weight prediction network to obtain a corresponding weight matrix. For example, the weight prediction network can use a normalization function to normalize the third feature vector, such as the normalization function can be softmax (normalized exponential function), the third feature vector is normalized to the interval [0, 1], and the corresponding weight matrix is obtained, and the sum of the elements included in the weight matrix is 1.
[0151] Based on the above processing, the third pixel matrix is input into the pre-trained enhancement parameter determination model, and the offset matrix and weight matrix output by the enhancement parameter determination model can be obtained, and the obtained offset matrix and weight matrix have higher accuracy; downsampling the third pixel matrix can increase the receptive field of the first feature extraction network and reduce the amount of calculation for subsequent processing, which can improve the efficiency of the offset matrix and weight matrix output by the enhancement parameter determination model. Subsequently, based on the offset matrix, the weight matrix, and the adjacent video frames of the reference video frame, the reference video frame is enhanced, which can improve the quality of the obtained target video and the efficiency of generating the target video.
[0152] For step S103, for each first image region in the reference video frame, the element corresponding to the first image region in the offset matrix represents the offset of the first image region relative to the adjacent video frame of the reference video frame. Further, according to the position of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame of the reference video frame, the second image region corresponding to the first image region in the adjacent video frame of the reference video frame is obtained.
[0153] In some embodiments, Figure 1 Based on Figure 3 , step S103 may include the following steps:
[0154] S1031: For each first image region in the reference video frame, obtain the coordinates of the first image region in the reference video frame.
[0155] S1032: Calculate the sum of the coordinates of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame as the reference coordinates.
[0156] S1033: Determine, in adjacent video frames, an image region corresponding to the reference coordinates, and obtain a second image region corresponding to the first image region in the adjacent video frames.
[0157] For each first image area in the reference video frame, the position of the first image area in the reference video frame can be represented by coordinates. For example, the electronic device can establish a coordinate system with the vertex in the upper left corner of the reference video frame as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. The first image area can be a rectangular area, and the electronic device can determine the coordinates of the four vertices of the first image area in the coordinate system, and the coordinates of the four vertices in the coordinate system can represent the position of the first image area in the reference video frame. Alternatively, the electronic device can determine the coordinates of the two vertices of the first image area in the coordinate system, as well as the length and width of the first image area, and the coordinates of the two vertices in the coordinate system, as well as the length and width of the first image area can represent the position of the first image area in the reference video frame.
[0158] In some embodiments, since the offset matrix output by the pre-trained enhanced parameter model is obtained through normalization, that is, each element in the offset matrix cannot indicate how many pixels the first image region is offset from the adjacent video frame of the reference video frame. Therefore, the electronic device may first restore the elements in the offset matrix so that the restored elements can indicate how many pixels the first image region is offset from the adjacent video frame of the reference video frame, that is, obtain the actual offset of the first image region from the adjacent video frame of the reference video frame.
[0159] Exemplarily, for each first image area in a reference video frame, the electronic device may calculate the product of the offset of the first image area relative to an adjacent video frame of the reference video frame and a third number to obtain an actual offset of the first image area relative to the adjacent video frame of the reference video frame, including an actual offset of the first image area in a horizontal direction relative to the adjacent video frame of the reference video frame, and an actual offset of the first image area in a vertical direction relative to the adjacent video frame of the reference video frame.
[0160] For example, if the motion range of a first image region is a circle with a radius of 64 pixels, the third number may be 64. If the offset of a first image region relative to an adjacent video frame of a reference video frame is 1, it means that the first image region moves a distance of 64 pixels relative to the adjacent video frame of the reference video frame, that is, the actual offset of the first image region relative to the adjacent video frame of the reference video frame is 64.
[0161] Then, the electronic device can calculate the horizontal coordinate of the first image area in the reference video frame and the sum of the actual offset of the first image area in the horizontal direction relative to the adjacent video frame of the reference video frame to obtain the horizontal coordinate of the reference coordinate corresponding to the first image area; calculate the vertical coordinate of the first image area in the reference video frame and the sum of the actual offset of the first image area in the vertical direction relative to the adjacent video frame of the reference video frame to obtain the vertical coordinate of the reference coordinate corresponding to the first image area.
[0162] For example, for each first image area in the reference video frame, the position of the first image area in the reference video frame can be expressed as: (1, 1), (1, 2), (2, 1), (2, 2); if the actual offset of the first image area in the horizontal direction relative to the adjacent video frame of the reference video frame is 1, and the actual offset in the vertical direction is 2, then the horizontal coordinate of the coordinates of the first image area in the reference video frame and the sum of the actual offset of the first image area in the horizontal direction relative to the adjacent video frame of the reference video frame are calculated, and the vertical coordinate of the coordinates of the first image area in the reference video frame and the sum of the actual offset of the first image area in the vertical direction relative to the adjacent video frame of the reference video frame are calculated to obtain the reference coordinates (2, 3), (2, 4), (3, 3), (3, 4) corresponding to the first image area.
[0163] Subsequently, the electronic device determines an image area corresponding to the reference coordinates corresponding to the first image area in an adjacent video frame of the reference video frame, and obtains a second image area corresponding to the first image area in the adjacent video frame of the reference video frame.
[0164] Based on the above processing, compared with the registration method based on feature point matching in the prior art, feature points in two video frames are first extracted, such as SIFT (Scale-Invariant Feature Transform) feature points, and then matching calculations are performed on the feature points in the two video frames to obtain feature points in one video frame corresponding to feature points in another video frame. According to the correspondence between the feature points in the two video frames, a transformation matrix is obtained, and then the transformation matrix is used to transform one video frame to achieve registration of the two video frames. Alternatively, a registration method based on block-matching technology calculates, for each image area in a video frame, the SAD (Sum of Absolute Difference) of the pixel value of each pixel point in the image area and the pixel value of each image area in the preset search area in the other video frame by means of a sliding window, and determines the image area with the smallest SAD in the other video frame as the image area corresponding to the image area in one video frame, and then obtains a transformation matrix based on the correspondence between the image areas in the two video frames, and then uses the transformation matrix to transform one video frame to achieve registration of the two video frames.
[0165] In the embodiment of the present invention, the offset matrix and the weight matrix are obtained by determining the model through the pre-trained enhancement parameters, and the accuracy of the obtained offset matrix and the weight matrix can be improved. And by restoring each element in the offset matrix, the restored element can represent the actual offset of the first image area relative to the adjacent video frame of the reference video frame, and the actual offset can represent the actual movement distance of the first image area relative to the adjacent video frame of the reference video frame, that is, the elements in the offset matrix can adapt to the reference video frame and the adjacent video frame of the reference video frame, and then determine the reference coordinates corresponding to the first image area according to the restored elements, which can improve the accuracy of the determined reference coordinates corresponding to the first image area.
[0166] For step S104, for each first image region in the reference video frame, the first image region and the second image region corresponding to the first image region in the adjacent video frame of the reference video frame contain the same content. If the reference video frame is located before the adjacent video frame of the reference video frame, the first image region is the image region before the second image region moves; if the reference video frame is located after the adjacent video frame of the reference video frame, the first image region is the image region after the second image region moves.
[0167] After determining the second image area corresponding to the first image area in the adjacent video frame of the reference video frame, the electronic device can enhance the first image area according to the weight of the first image area and the weight of the second image area to obtain a target video frame corresponding to the reference video frame.
[0168] In some embodiments, Figure 3 Based on Figure 4 , step S104 may include the following steps:
[0169] S1041: For each pixel point in the first image area in the reference video frame, if the reference coordinate corresponding to the pixel point in the adjacent video frame is an integer, calculate the weighted sum of the pixel value of the pixel point and the pixel value of the pixel point at the reference coordinate according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, and obtain the target video frame corresponding to the reference video frame.
[0170] S1042: If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, determine the pixel value corresponding to the reference coordinate based on the pixel values of the pixel points adjacent to the reference coordinate, and calculate the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
[0171] For each first image area in a reference video frame, the first image area includes multiple pixel points (which may be referred to as first pixel points), and the weights of the multiple first pixel points are the same, which are all the weights of the first image area; for each second image area in an adjacent video frame of the reference video frame, the second image area includes multiple pixel points (which may be referred to as second pixel points), and the weights of the multiple second pixel points are the same, which are all the weights of the second image area.
[0172] For each first pixel in the first image area, after the electronic device calculates the reference coordinate corresponding to the first pixel in the adjacent video frame of the reference video frame, if the reference coordinate corresponding to the first pixel in the adjacent video frame of the reference video frame is an integer, that is, the electronic device can determine the second pixel corresponding to the first pixel in the adjacent video frame of the reference video frame, then the electronic device can directly calculate the weighted sum of the pixel value of the first pixel and the pixel value of the second pixel corresponding to the first pixel according to the weight of the first image area to which the first pixel belongs and the weight of the second image area to which the second pixel corresponding to the first pixel belongs, to obtain the enhanced pixel value of the first image area. The second image area to which the second pixel corresponding to the first pixel belongs, that is, the second image area corresponding to the first image area to which the first pixel belongs in the adjacent video frame of the reference video frame. The electronic device calculates the enhanced pixel values of each first image area in the reference video frame, and can also obtain the enhanced reference video frame, that is, the target video frame.
[0173] If the reference coordinates corresponding to the first pixel in the adjacent video frame of the reference video frame are not integers, that is, the electronic device fails to determine the second pixel corresponding to the first pixel in the adjacent video frame of the reference video frame, then the electronic device cannot obtain the pixel value of the second pixel corresponding to the first pixel, and cannot calculate the enhanced pixel value of the first pixel based on the pixel value of the second pixel corresponding to the first pixel, and thus cannot enhance the reference video frame. Accordingly, the electronic device can determine the pixel value corresponding to the reference coordinate corresponding to the first pixel based on the pixel values of the pixels adjacent to the reference coordinates corresponding to the first pixel in the adjacent video frame of the reference video frame.
[0174] In some embodiments, Figure 4 Based on Figure 5 , step S1042 may include the following steps:
[0175] S10421: If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, based on the nearest neighbor interpolation algorithm, determine the pixel value of the pixel point with the smallest distance to the reference coordinate as the pixel value corresponding to the reference coordinate; or, based on the bilinear interpolation algorithm, calculate the pixel value corresponding to the reference coordinate based on the pixel values of a preset number of pixels within a preset neighborhood of the reference coordinate, and calculate the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
[0176] The pixel point adjacent to the reference coordinate corresponding to the first pixel point in the adjacent video frame of the reference video frame may be: the pixel point with the shortest distance to the reference coordinate corresponding to the first pixel point.
[0177] For example, if the reference coordinates corresponding to the first pixel point are (2.1, 2.2), then the integer coordinates with the smallest distance from the reference coordinates corresponding to the first pixel point are (2, 2). The electronic device can use the pixel value of the pixel point with the coordinates (2, 2) as the pixel value corresponding to the reference coordinates corresponding to the first pixel point.
[0178] The pixel points adjacent to the reference coordinates corresponding to the first pixel point in the adjacent video frame of the reference video frame may be: a preset number of pixel points within a preset neighborhood range of the reference coordinates corresponding to the first pixel point that are closest to the reference coordinates corresponding to the first pixel point. For example, the preset number may be 4.
[0179] For example, if the reference coordinates corresponding to the first pixel point are (12.5, 13.5), the preset neighborhood range of the reference coordinates corresponding to the first pixel point is: a circular area with the reference coordinates corresponding to the first pixel point as the center and a radius of 1. Then the pixel points adjacent to the reference coordinates corresponding to the first pixel point are: the pixel point with coordinates (12, 13), the pixel point with coordinates (12, 14), the pixel point with coordinates (13, 13), and the pixel point with coordinates (13, 14).
[0180] Furthermore, the electronic device uses a bilinear interpolation algorithm to perform bilinear interpolation on the pixel values of a preset number of pixel points adjacent to the reference coordinate corresponding to the first pixel point, to obtain the pixel value corresponding to the reference coordinate corresponding to the first pixel point.
[0181] Subsequently, the electronic device may calculate the weighted sum of the pixel value of the first pixel and the pixel value corresponding to the reference coordinate corresponding to the first pixel according to the weight of the first image area to which the first pixel belongs and the weight of the second image area to which the reference coordinate corresponding to the first pixel belongs, to obtain the enhanced pixel value of the first image area. The electronic device calculates the enhanced pixel values of each first image area in the reference video frame, and thus obtains the enhanced reference video frame, i.e., the target video frame.
[0182] Based on the above processing, if the reference coordinate corresponding to the first pixel is an integer, that is, the electronic device can determine the second pixel corresponding to the first pixel in the adjacent video frame of the reference video frame, then the electronic device directly calculates the enhanced pixel value of the first pixel according to the pixel value of the second pixel corresponding to the first pixel; if the reference coordinate corresponding to the first pixel is not an integer, that is, the electronic device cannot determine the second pixel corresponding to the first pixel in the adjacent video frame of the reference video frame, then the electronic device determines the pixel value corresponding to the reference coordinate corresponding to the first pixel based on the nearest neighbor interpolation algorithm or the bilinear interpolation algorithm according to the pixel values of the pixels adjacent to the reference coordinate corresponding to the first pixel, and then calculates the enhanced pixel value of the first pixel according to the pixel value corresponding to the reference coordinate corresponding to the first pixel. The accuracy of the calculated enhanced pixel value can be improved, and the accuracy of the target video frame corresponding to the obtained reference video frame can be improved.
[0183] For step S105, after the electronic device obtains the target video frames corresponding to the designated reference video frames in the original video, the reference video frames in the original video are enhanced. Subsequently, the electronic device can combine the target video frames corresponding to the reference video frames to obtain the target video corresponding to the original video.
[0184] Exemplarily, the designated reference video frames in the original video may have a preset order. After the electronic device obtains the target video frames corresponding to the reference video frames, it sorts the target video frames corresponding to the reference video frames according to the preset order between the reference video frames, and then splices the target video frames corresponding to the sorted reference video frames into one video to obtain the target video.
[0185] See also Figure 6 , Figure 6 The first flow chart of the model training method provided in the embodiment of the present invention is used to generate the enhanced parameter determination model described in any of the above embodiments. The method may include the following steps:
[0186] S601: Acquire a first sample video frame and a second sample video frame.
[0187] The first sample video frame and the second sample video frame are RAW images; the first sample video frame is obtained by performing noise addition processing on the second sample video frame.
[0188] S602: Input the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, and obtain the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure.
[0189] Among them, the elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the first sample video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area.
[0190] S603: For each first image region in the first sample video frame, determine a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the first sample video frame and an offset of the first image region relative to an adjacent video frame.
[0191] S604: For each first image region in the first sample video frame, according to the weight of the first image region and the weight of the second image region corresponding to the first image region in the adjacent video frame, calculate the enhanced pixel value of the first image region to obtain a predicted video frame corresponding to the first sample video frame.
[0192] S605: Calculate a loss function value representing the difference between the predicted video frame and the second sample video frame.
[0193] S606: Adjust the model parameters of the enhancement parameter determination model of the initial structure based on the calculated loss function value until a preset convergence condition is reached to obtain a trained enhancement parameter determination model.
[0194] Based on the model training method provided by the embodiment of the present invention, an enhancement parameter determination model can be obtained. Subsequently, based on the pre-trained enhancement parameter determination model, the offset matrix and weight matrix corresponding to the reference video frame input to the enhancement parameter determination model and the adjacent video frame of the reference video frame can be obtained. Subsequently, the reference video frame can be enhanced according to the offset matrix and the weight matrix. The reference video frame is a RAW image, that is, the RAW image can be enhanced; based on the pre-trained enhancement parameter determination model, the offset matrix and the weight matrix are obtained, then for each first image area in the reference video frame, the second image area corresponding to the first image area in the adjacent video frame of the reference video frame is determined based on the offset matrix, which can improve the accuracy of the determined second image area. Furthermore, based on the weights of each first image area in the reference video frame and the weights of the second image area corresponding to each first image area, the pair of reference video frames is enhanced, which can improve the quality of the obtained target video.
[0195] For step S601, the first sample video frame is obtained by adding noise to the second sample video frame. For example, under normal illumination, using the minimum ISO (sensitivity) of the camera, a technician holds the camera to simulate the movement of the camera lens and shoots the video to be processed. Subsequently, the electronic device can determine the first sample video frame, the second sample video frame and the sample video according to the video to be processed.
[0196] In one implementation, each video frame in the video to be processed is a second sample video frame. The electronic device directly performs noise addition processing on each video frame in the video to be processed to obtain a sample video, and the video frame in the sample video corresponding to the second sample video frame in the video to be processed is the first sample video frame corresponding to the second sample video frame.
[0197] In another implementation, the electronic device performs illumination reduction processing on each video frame in the video to be processed to obtain a video frame in a low illumination environment (which may be called a low illumination video frame) corresponding to each video frame in the video to be processed, thereby obtaining a video in a low illumination environment corresponding to the video to be processed (which may be called a low illumination video), and uses the low illumination video frame as the second sample video frame.
[0198] Exemplarily, for each video frame in the video to be processed, the electronic device can calculate the quotient of the pixel value of each pixel point of the video frame and the preset ratio (ratio) to obtain a low-light video frame corresponding to the video frame, and a low-light video corresponding to the video to be processed can be obtained. The preset ratio can be 10, 20, 30, etc. The pixel value of each pixel point of the second sample video frame can be recorded as dark light RAW domain data, or dark light GT data.
[0199] Then, the electronic device can perform noise addition processing on each video frame in the low-illumination video to obtain a sample video, wherein the video frame in the sample video corresponding to the second sample video frame in the low-illumination video is the first sample video frame corresponding to the second sample video frame.
[0200] For step S602, the adjacent video frames of the first sample video frame in the sample video to which it belongs include: a first number of video frames in the sample video that are located before the first sample video frame, and / or a second number of video frames that are located after the first sample video frame. The way in which the electronic device determines the adjacent video frames of the first sample video frame in the sample video to which it belongs is similar to the way in which the electronic device determines the adjacent video frames of the reference video frame in the aforementioned embodiment, and reference may be made to the relevant description of the aforementioned embodiment.
[0201] The enhancement parameter determination model of the initial structure may be a CNN model. After acquiring the first sample video frame, the electronic device may obtain a pixel matrix (which may be referred to as a fourth pixel matrix) composed of pixel values of each pixel point of the first sample video frame. After acquiring the adjacent video frame of the first sample video frame, the electronic device may obtain a pixel matrix (which may be referred to as a fifth pixel matrix) of the pixel values of each pixel point of the adjacent video frame. Then, the electronic device may splice the fourth pixel matrix with the fifth pixel matrix to obtain a spliced pixel matrix (which may be referred to as a sixth pixel matrix).
[0202] Exemplarily, the pixel matrix composed of the pixel values of each pixel point of each video frame in the video to be processed can be the RAW domain data of the video frame. After the electronic device obtains the video to be processed, every 5 consecutive video frames in the video to be processed are used as a minimum processing unit. For each minimum processing unit in the video to be processed, the 5 consecutive video frames included in the minimum processing unit are respectively recorded as: alt_img0 video frame, alt_img1 video frame, ref_img video frame, alt_img2 video frame, alt_img3 video frame.
[0203] For each video frame in the minimum processing unit, the video frame is subjected to illumination reduction processing, that is, the RAW domain data of the video frame is divided by a preset ratio to obtain the dark light RAW domain data corresponding to the video frame, that is, to obtain the low illumination video frame corresponding to the video frame, and then to obtain the low illumination processing unit corresponding to the minimum processing unit. For example, the alt_img0 video frame is subjected to illumination reduction processing to obtain the low illumination video frame corresponding to the alt_img0 video frame, which is recorded as gt_alt_img0 video frame; the ref_img video frame is subjected to illumination reduction processing to obtain the low illumination video frame corresponding to the ref_img video frame, which is recorded as gt_ref_img video frame. The electronic device can select the low illumination video frame corresponding to the ref_img video frame, that is, the gt_ref_img video frame, as the second sample video frame.
[0204] Then, the electronic device can add preset noise to each low illumination video frame in the low illumination processing unit to obtain the low illumination video frame after noise addition corresponding to each low illumination video frame in the low illumination processing unit, and then obtain the sample video. Each video frame in the sample video is a low illumination video frame after noise addition corresponding to each video frame in the video to be processed. For example, a preset noise is added to the low illumination video frame gt_alt_img0 video frame corresponding to the alt_img0 video frame to obtain the corresponding low illumination video frame after noise addition, which is recorded as noise_alt_img0 video frame; a preset noise is added to the low illumination video frame gt_ref_img video frame corresponding to the ref_img video frame to obtain the corresponding low illumination video frame after noise addition, which is recorded as noise_ref_img video frame. Then, the electronic device can select the low illumination video frame after noise addition corresponding to the ref_img video frame as the first sample video frame, that is, the noise_ref_img video frame is used as the first sample video frame corresponding to the second sample video frame. Correspondingly, the adjacent video frames of the first sample video frame include: noise_alt_img0 video frame, noise_alt_img1 video frame, noise_alt_img2 video frame, and noise_alt_img3 video frame.
[0205] Then, the electronic device splices the noisy low-illumination video frames corresponding to the five consecutive video frames included in the minimum processing unit. The splicing result can be recorded as {noise_alt_img0; noise_alt_img1; noise_ref_img; noise_alt_img2, noise_alt_img3}, which serves as the input data of the enhanced parameter determination model of the initial structure.
[0206] Subsequently, the electronic device may input the sixth pixel matrix into the enhancement parameter determination model of the initial structure to obtain the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure. The elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame of the first sample video frame; the elements in the weight matrix correspond one-to-one with the first image area in the first sample video frame and the second image area in the adjacent video frame of the first sample video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame of the first sample video frame, the element represents the weight of the corresponding second image area.
[0207] In some embodiments, the enhanced parameter determination model of the initial structure includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network, and a weight prediction network.
[0208] Accordingly, in Figure 6 Based on Figure 7 , step S602 includes the following steps:
[0209] S6021: Input the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, perform feature extraction on the first sample video frame and the adjacent video frames through the first feature extraction network, and obtain a feature vector representing the image features of the first sample video frame and the image features of the adjacent video frames as the first feature vector.
[0210] S6022: Perform feature extraction on the first feature vector through a second feature extraction network to obtain a second feature vector.
[0211] S6023: Normalize the second eigenvector through the offset prediction network to obtain a corresponding offset matrix.
[0212] S6024: Perform feature extraction on the first feature vector through a third feature extraction network to obtain a third feature vector.
[0213] S6025: Normalize the third eigenvector through the weight prediction network to obtain a corresponding weight matrix.
[0214] The first feature extraction network may include multiple convolutional layers, including: a convolutional layer with a convolution kernel of 3×3 and a stride of 1, and a convolutional layer with a convolution kernel of 3×3 and a stride of 2. In addition, each convolutional layer included in the first feature extraction network is also connected to an activation function, such as a ReLu function.
[0215] The sixth pixel matrix is subjected to feature extraction through the multiple convolutional layers included in the first feature extraction network, and downsampling of the sixth pixel matrix is achieved to obtain a feature vector (i.e., a first feature vector) representing image features of the first sample video frame and image features of adjacent video frames of the first sample video frame.
[0216] The second feature extraction network may include two convolution layers, the two convolution layers are convolution layers with a convolution kernel of 3×3 and a step size of 1, and the first convolution layer in the second feature extraction network is connected with an activation function, such as a ReLu function; since each first image region in the first sample video frame has two offsets relative to each adjacent video frame of the first sample video frame, that is, each adjacent video frame of the first sample video frame corresponds to two output channels of the second feature extraction network. Therefore, the number of output channels of the second feature extraction network is twice the number of adjacent video frames of the first sample video frame.
[0217] The first feature vector is feature extracted by two convolutional layers included in the second feature extraction network to obtain a feature vector (i.e., a second feature vector) representing the offset of the first sample video frame relative to the adjacent video frame of the first sample video frame. Subsequently, the second feature vector is normalized by the offset prediction network to obtain a corresponding offset matrix. For example, the offset prediction network can use a normalization function to normalize the second feature vector, such as the normalization function can be a tanh function, and the second feature vector is normalized to the interval [-1, 1] to obtain a corresponding offset matrix.
[0218] The third feature extraction network may include two convolutional layers, wherein the two convolutional layers are convolutional layers with a convolution kernel of 3×3 and a step size of 1, and an activation function is connected to the first convolutional layer in the third feature extraction network, such as the activation function may be a ReLu function; since the first sample video frame corresponds to an output channel of the third feature extraction network, and each adjacent video frame of the first sample video frame also corresponds to an output channel of the third feature extraction network, the number of output channels of the third feature extraction network is the same as the sum of the number of the first sample video frames and the number of adjacent video frames of the first sample video frame.
[0219] The first feature vector is feature extracted by two convolutional layers included in the third feature extraction network to obtain a feature vector (i.e., a third feature vector) representing the weights of the first sample video frame and the adjacent video frames of the first sample video frame. Subsequently, the third feature vector is normalized by the weight prediction network to obtain a corresponding weight matrix. For example, the weight prediction network can use a normalization function to normalize the third feature vector, such as the normalization function can be softmax, and the third feature vector is normalized to the interval [0, 1] to obtain a corresponding weight matrix, and the sum of the elements included in the weight matrix is 1.
[0220] Based on the above processing, the sixth pixel matrix is input into the enhancement parameter determination model of the initial structure, and the offset matrix and weight matrix output by the enhancement parameter determination model can be obtained. The first sample video frame is subsequently enhanced according to the offset matrix and the weight matrix, and the parameters of the enhancement parameter determination model are adjusted according to the enhancement result, which can improve the accuracy of the offset matrix and the weight matrix output by the enhancement parameter determination model. Downsampling the sixth pixel matrix can increase the receptive field of the first feature extraction network and reduce the amount of calculation for subsequent processing, which can improve the efficiency of the offset matrix and the weight matrix output by the enhancement parameter determination model.
[0221] With respect to step S603 , for each first image region in the first sample video frame, the position of the first image region in the first sample video frame may be represented by coordinates.
[0222] In some embodiments, since the offset matrix output by the enhanced parameter model of the initial structure is obtained through normalization processing, that is, each element in the offset matrix cannot directly indicate how many pixels the first image area is offset relative to the adjacent video frame of the first sample video frame. Therefore, the electronic device can first restore the elements in the offset matrix so that the restored elements can indicate how many pixels the first image area is offset relative to the adjacent video frame of the first sample video frame, that is, obtain the actual offset of the first image area relative to the adjacent video frame of the first sample video frame, and the actual offset can indicate the actual movement distance of the first image area relative to the adjacent video frame of the first sample video frame. The actual offset of the first image area relative to the adjacent video frame of the first sample video frame includes: the actual offset of the first image area relative to the adjacent video frame of the first sample video frame in the horizontal direction, and the actual offset of the first image area relative to the adjacent video frame of the first sample video frame in the vertical direction.
[0223] Then, the electronic device can calculate the sum of the horizontal coordinate of the first image area in the first sample video frame and the actual offset of the first image area in the horizontal direction relative to the adjacent video frame of the first sample video frame to obtain the horizontal coordinate of the reference coordinate corresponding to the first image area; calculate the vertical coordinate of the first image area in the first sample video frame and the actual offset of the first image area in the vertical direction relative to the adjacent video frame of the first sample video frame to obtain the vertical coordinate of the reference coordinate corresponding to the first image area. That is, the second image area corresponding to the first image area in the adjacent video frame of the first sample video frame can be determined.
[0224] For step S604, the electronic device calculates the enhanced pixel value of each first image area in the first sample video frame to obtain the predicted video frame corresponding to the first sample video frame. This is similar to the way in which the electronic device calculates the enhanced pixel value of each first image area in the reference video frame to obtain the target video frame corresponding to the reference video frame in the aforementioned embodiment. Please refer to the relevant introduction of the aforementioned embodiment.
[0225] Exemplarily, the electronic device may enhance the first sample video frame based on the following formula (1), based on the weights of each first image area in the first sample video frame, and the weights of the second image areas corresponding to each first image area in the adjacent video frame of the first sample video frame, to obtain a predicted video frame corresponding to the first sample video frame.
[0226]
[0227] Among them, out represents the predicted video frame, noise_ref m represents the mth first image region in the first sample video frame, wts_refm Indicates the weight of the mth first image region, warped_noise_alt m [i] represents the second image area corresponding to the mth first image area in the i-th adjacent video frame, wts_alt m [i] represents the weight of the second image region corresponding to the mth first image region in the ith adjacent video frame, and j represents that the first sample video frame has j+1 adjacent video frames. For example, if the first sample video frame has 4 adjacent video frames, then j in the above formula (1) is 3.
[0228] With respect to step S605 and step S606, after obtaining the predicted video frame corresponding to the first sample video frame, the electronic device may calculate the difference between the predicted video frame and the second sample video frame.
[0229] The loss function may be an absolute difference loss function, and the electronic device may calculate an absolute difference loss function value representing the difference between the predicted video frame and the second sample video frame. Exemplarily, the electronic device may calculate an absolute difference loss function value representing the difference between the predicted video frame and the second sample video frame based on the following formula (2):
[0230] L1_loss=|out-ref_img| (2)
[0231] Among them, L1_loss represents the absolute difference loss function value, out represents the predicted video frame, and ref_img represents the second sample video frame.
[0232] Then, the electronic device adjusts the model parameters of the enhancement parameter determination model of the initial structure based on the calculated loss function value, for example, by adjusting the model parameters of the enhancement parameter determination model of the initial structure in a gradient descent manner until a preset convergence condition is reached, thereby obtaining a trained enhancement parameter determination model. The process of training the enhancement parameter determination model of the initial structure can also be called a supervised learning process.
[0233] See also Figure 8 , Figure 8 A working principle diagram of the enhanced parameter determination model provided in an embodiment of the present invention.
[0234] Figure 8 The input in is the input data of the enhancement parameter determination model, and the input data is the first sample video frame and the adjacent video frames of the first sample video frame.
[0235] Figure 8In the figure, the first type of convolutional layer is represented by a black rectangle, and the first type of convolutional layer is denoted by conv3×3_stride=1+relu. The convolution kernel of the first type of convolutional layer is 3×3, the stride is 1, and an activation function is connected. The second type of convolutional layer is represented by a white rectangle, and the second type of convolutional layer is denoted by conv3×3_stride=2+relu. The convolution kernel of the second type of convolutional layer is 3×3, the stride is 2, and an activation function is connected.
[0236] The first feature extraction network includes nine convolutional layers, of which the first three convolutional layers, the fifth convolutional layer, the sixth convolutional layer, the eighth convolutional layer and the ninth convolutional layer are all first-type convolutional layers; and the fourth convolutional layer and the seventh convolutional layer are second-type convolutional layers. The nine convolutional layers included in the first feature extraction network are used to extract features of the input data, downsample the input data, and obtain a first feature vector representing image features of the first sample video frame and image features of adjacent video frames of the first sample video frame.
[0237] Figure 8 In the figure, a black rectangle filled with white horizontal stripes represents the third type of convolutional layer. The third type of convolutional layer is denoted as conv3×3_stride=1, c=8. The convolution kernel of the third type of convolutional layer is 3×3, and the stride is 1.
[0238] The second feature extraction network includes two convolutional layers, the first convolutional layer of the two convolutional layers is a first type of convolutional layer, and the second convolutional layer of the two convolutional layers is a third type of convolutional layer. The first feature vector is feature extracted by the two convolutional layers included in the second feature extraction network to obtain a second feature vector representing an offset of the first sample video frame relative to an adjacent video frame of the first sample video frame.
[0239] Figure 8 The offset prediction network is represented by a black rectangle filled with white diagonal stripes. The offset prediction network can be denoted as tanh. The offset prediction network can use a normalization function to normalize the second eigenvector. For example, the normalization function can be a tanh function, and the second eigenvector is normalized to the interval [-1, 1] to obtain the corresponding offset matrix. The output of the offset prediction network can be denoted as out_flow, c=8 (output_offset, the number of output channels is 8). The output of the offset prediction network can also be called motion vectors.
[0240] Figure 8 In the figure, a white rectangle filled with black horizontal stripes represents the fourth type of convolutional layer. The fourth type of convolutional layer is denoted as conv3×3_stride=1, c=5. The convolution kernel of the fourth type of convolutional layer is 3×3, and the stride is 1.
[0241] The third feature extraction network includes two convolutional layers, the first convolutional layer of the two convolutional layers is a first type of convolutional layer, and the second convolutional layer of the two convolutional layers is a fourth type of convolutional layer. The first feature vector is feature extracted by the two convolutional layers included in the third feature extraction network to obtain a third feature vector representing the weights of the first sample video frame and the adjacent video frames of the first sample video frame.
[0242] Figure 8 In the figure, the weight prediction network is represented by a white rectangle filled with black diagonal stripes, and the weight prediction network can be recorded as softmax. The weight prediction network can use a normalization function to normalize the third eigenvector, such as softmax, normalize the third eigenvector to the interval [0, 1], and obtain the corresponding weight matrix, and the sum of the elements included in the weight matrix is 1. The output of the weight prediction network can be recorded as out_wts, c=5 (output_weight, the number of output channels is 5). The output of the weight prediction network can also be called a fusion weight.
[0243] Furthermore, the electronic device can enhance the first sample video frame based on the formula (1) in the aforementioned embodiment, based on the weights of each first image region in the first sample video frame, and the weights of the second image regions corresponding to each first image region in the adjacent video frame of the first sample video frame, to obtain a predicted video frame corresponding to the first sample video frame. Then, the electronic device can calculate the absolute difference loss function value representing the difference between the predicted video frame and the second sample video frame based on the formula (2) in the aforementioned embodiment, and adjust the model parameters of the enhanced parameter determination model of the initial structure based on the calculated absolute difference loss function value until the preset convergence condition is reached, thereby obtaining a trained enhanced parameter determination model. Subsequently, the electronic device can input 5 consecutive video frames in the sample video into the trained enhanced parameter determination model, and the enhanced parameter determination model can perform a forward reasoning process to obtain an enhanced video image output.
[0244] Based on the above processing, an enhancement parameter determination model can be obtained. Subsequently, based on the pre-trained enhancement parameter determination model, the offset matrix and weight matrix corresponding to the reference video frame input to the enhancement parameter determination model and the adjacent video frame of the reference video frame can be obtained. Subsequently, the reference video frame can be enhanced according to the offset matrix and the weight matrix. The reference video frame is a RAW image, that is, the RAW image can be enhanced; based on the pre-trained enhancement parameter determination model, the offset matrix and the weight matrix are obtained, then for each first image area in the reference video frame, the second image area corresponding to the first image area in the adjacent video frame of the reference video frame is determined based on the offset matrix, which can improve the accuracy of the determined second image area. Furthermore, based on the weights of each first image area in the reference video frame and the weights of the second image area corresponding to each first image area, the pair of reference video frames is enhanced, which can improve the quality of the obtained target video.
[0245] Moreover, by enhancing the parameters to determine the model, it is possible to output offsets and weights with higher accuracy, which simplifies the subsequent processing steps, enables real-time registration enhancement of RAW images, and achieves better results in time domain enhancement of original videos.
[0246] In addition, the model directly outputs the offset and weight between the RAW images based on the enhancement parameters, and the time domain enhancement effect is better. The subsequent processing is simple, and real-time operation can be achieved on a general NPU (Neural-network Process Units, embedded neural network processor) platform.
[0247] See also Fig. 9 , Fig. 9 A comparative example diagram of the video enhancement effect provided by an embodiment of the present invention. Fig. 9 The left image in is a reference video frame, and there are multiple noise points in the reference video frame, that is, the clarity of the reference video frame is low; Fig. 9 The intermediate image in the image is an enhanced video frame (which may be referred to as a comparison video frame) obtained by enhancing the reference video frame based on the prior art. The definition of the comparison video frame is also low and there is a ghosting problem. Fig. 9 The right image in is a target video frame obtained by the video enhancement method provided by an embodiment of the present invention. Compared with the reference video frame and the comparison video frame, the target video frame has fewer noise points and clearer details. It can be seen that since the video enhancement method provided by an embodiment of the present invention can perform time domain enhancement on the reference video frame, the RAW image can be enhanced; and the clarity of the obtained target video can be improved, that is, the quality of the obtained target video can be improved.
[0248] Based on the same inventive concept as the above-mentioned video enhancement method, an embodiment of the present invention further provides a video enhancement device. Fig.10 , Fig.10 A structural diagram of a video enhancement device provided by an embodiment of the present invention, the device includes:
[0249] The reference video frame determination module 1001 is used to determine a designated reference video frame from each video frame of the original video; wherein each video frame in the original video is a RAW image;
[0250] A matrix acquisition module 1002 is used to input the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model;
[0251] The elements in the offset matrix represent the offset of the first image region in the reference video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image region in the reference video frame and the second image region in the adjacent video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region; if an element corresponds to the second image region in the adjacent video frame, the element represents the weight of the corresponding second image region;
[0252] The image region determination module 1003 is configured to determine, for each first image region in the reference video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame;
[0253] The target video frame acquisition module 1004 is used to calculate, for each first image region in the reference video frame, an enhanced pixel value of the first image region according to the weight of the first image region and the weight of the second image region corresponding to the first image region in the adjacent video frame, so as to obtain a target video frame corresponding to the reference video frame;
[0254] The target video acquisition module 1005 is used to obtain the target video corresponding to the original video based on the target video frame.
[0255] Optionally, the enhanced parameter determination model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network;
[0256] The matrix acquisition module 1002 is specifically used for:
[0257] Inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model, performing feature extraction on the reference video frame and the adjacent video frame through the first feature extraction network, and obtaining a feature vector representing the image features of the reference video frame and the image features of the adjacent video frame as a first feature vector;
[0258] Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector;
[0259] Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix;
[0260] Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector;
[0261] The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
[0262] Optionally, the image area determination module 1003 is specifically configured to:
[0263] For each first image region in the reference video frame, obtaining the coordinates of the first image region in the reference video frame;
[0264] Calculating a sum of the coordinates of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame as a reference coordinate;
[0265] In the adjacent video frames, an image region corresponding to the reference coordinates is determined to obtain a second image region corresponding to the first image region in the adjacent video frames.
[0266] Optionally, the target video frame acquisition module 1004 is specifically used to:
[0267] For each pixel point in the first image area in the reference video frame, if the reference coordinate corresponding to the pixel point in the adjacent video frame is an integer, according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, a weighted sum of the pixel value of the pixel point and the pixel value of the pixel point at the reference coordinate is calculated to obtain a target video frame corresponding to the reference video frame;
[0268] If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, the pixel value corresponding to the reference coordinate is determined based on the pixel values of the pixel points adjacent to the reference coordinate, and the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate is calculated according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
[0269] Optionally, the target video frame acquisition module 1004 is specifically used to:
[0270] Based on a nearest neighbor interpolation algorithm, determine the pixel value of the pixel point with the smallest distance from the reference coordinate as the pixel value corresponding to the reference coordinate;
[0271] or,
[0272] Based on a bilinear interpolation algorithm, the pixel value corresponding to the reference coordinate is calculated based on the pixel values of a preset number of pixel points within a preset neighborhood range of the reference coordinate.
[0273] Optionally, the adjacent video frames of the reference video frame include: a video frame located before the reference video frame in the original video, and / or a video frame located after the reference video frame.
[0274] Based on the video enhancement device provided by the embodiment of the present invention, the original video including each video frame as a RAW image can be enhanced based on the offset matrix and the weight matrix to obtain the target video corresponding to the original video, that is, the RAW image can be enhanced; and, since the offset matrix and the weight matrix are obtained based on the pre-trained enhancement parameter determination model, for each first image area in the reference video frame, the second image area corresponding to the first image area in the adjacent video frame of the reference video frame is determined based on the offset matrix, which can improve the accuracy of the determined second image area. Furthermore, based on the weights of each first image area in the reference video frame and the weights of the second image area corresponding to each first image area, the pair of reference video frames is enhanced to improve the quality of the obtained target video.
[0275] Based on the same inventive concept as the above-mentioned model training method, an embodiment of the present invention further provides a model training device for generating the enhanced parameter determination model described in any of the above-mentioned embodiments. Fig.11 , Fig.11 A structural diagram of a model training device provided in an embodiment of the present invention, the device includes:
[0276] The sample video frame acquisition module 1101 is used to acquire a first sample video frame and a second sample video frame; wherein the first sample video frame and the second sample video frame are RAW images; and the first sample video frame is obtained by performing noise addition processing on the second sample video frame;
[0277] A matrix acquisition module 1102 is used to input the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, and obtain the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure;
[0278] The elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the first sample video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area;
[0279] An image region determining module 1103 is configured to determine, for each first image region in the first sample video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the first sample video frame and an offset of the first image region relative to the adjacent video frame;
[0280] The predicted video frame acquisition module 1104 is configured to calculate, for each first image region in the first sample video frame, an enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, so as to obtain a predicted video frame corresponding to the first sample video frame;
[0281] a difference calculation module 1105, configured to calculate a loss function value representing a difference between the predicted video frame and the second sample video frame;
[0282] The adjustment module 1106 is used to adjust the model parameters of the enhancement parameter determination model of the initial structure based on the calculated loss function value until a preset convergence condition is reached to obtain a trained enhancement parameter determination model.
[0283] Optionally, the enhancement parameter determination model of the initial structure includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network;
[0284] The matrix acquisition module 1102 is specifically used for:
[0285] Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, performing feature extraction on the first sample video frame and the adjacent video frames through the first feature extraction network, and obtaining a feature vector representing the image features of the first sample video frame and the image features of the adjacent video frames as a first feature vector;
[0286] Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector;
[0287] Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix;
[0288] Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector;
[0289] The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
[0290] Based on the model training device provided by the embodiment of the present invention, an enhancement parameter determination model can be obtained. Subsequently, based on the pre-trained enhancement parameter determination model, the offset matrix and weight matrix corresponding to the reference video frame input to the enhancement parameter determination model and the adjacent video frame of the reference video frame can be obtained. Subsequently, the reference video frame can be enhanced according to the offset matrix and the weight matrix. The reference video frame is a RAW image, that is, the RAW image can be enhanced; based on the pre-trained enhancement parameter determination model, the offset matrix and the weight matrix are obtained, then for each first image area in the reference video frame, the second image area corresponding to the first image area in the adjacent video frame of the reference video frame is determined based on the offset matrix, which can improve the accuracy of the determined second image area. Furthermore, based on the weights of each first image area in the reference video frame and the weights of the second image area corresponding to each first image area, the pair of reference video frames is enhanced, which can improve the quality of the obtained target video.
[0291] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.
[0292] Memory 1203, used for storing computer programs;
[0293] The processor 1201 is used to implement the steps of any video enhancement method in the above embodiments when executing the program stored in the memory 1203.
[0294] The embodiment of the present invention further provides an electronic device, such as Fig.13 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.
[0295] Memory 1303, used for storing computer programs;
[0296] Processor 1301 is used to implement the steps of any model training method in the above embodiments when executing the program stored in memory 1303.
[0297] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0298] The communication interface is used for communication between the above electronic device and other devices.
[0299] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0300] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0301] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned video enhancement methods are implemented.
[0302] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods are implemented.
[0303] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the video enhancement methods in the above embodiments.
[0304] In another embodiment provided by the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method in the above embodiments.
[0305] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0306] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0307] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0308] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A video enhancement method, It is characterized in that The method comprises: Determine a designated reference video frame from each video frame of the original video; wherein each video frame in the original video is a RAW image; Inputting the reference video frame and adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model; The elements in the offset matrix represent the offset of the first image region in the reference video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image region in the reference video frame and the second image region in the adjacent video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region; if an element corresponds to the second image region in the adjacent video frame, the element represents the weight of the corresponding second image region; For each first image region in the reference video frame, based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame, determining a second image region corresponding to the first image region in the adjacent video frame; For each first image region in the reference video frame, according to the weight of the first image region and the weight of the second image region corresponding to the first image region in the adjacent video frame, the enhanced pixel value of the first image region is calculated to obtain a target video frame corresponding to the reference video frame; Based on the target video frame, a target video corresponding to the original video is obtained.
2. The method according to claim 1, It is characterized in that The enhanced parameter determination model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network; The step of inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model includes: Inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model, performing feature extraction on the reference video frame and the adjacent video frame through the first feature extraction network, and obtaining a feature vector representing the image features of the reference video frame and the image features of the adjacent video frame as a first feature vector; Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector; Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix; Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector; The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
3. The method according to claim 1, It is characterized in that The determining, for each first image region in the reference video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame, comprises: For each first image region in the reference video frame, obtaining the coordinates of the first image region in the reference video frame; Calculating a sum of the coordinates of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame as a reference coordinate; In the adjacent video frames, an image region corresponding to the reference coordinates is determined to obtain a second image region corresponding to the first image region in the adjacent video frames.
4. The method according to claim 3, It is characterized in that The step of calculating, for each first image region in the reference video frame, an enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame to obtain a target video frame corresponding to the reference video frame includes: For each pixel point in the first image area in the reference video frame, if the reference coordinate corresponding to the pixel point in the adjacent video frame is an integer, according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, a weighted sum of the pixel value of the pixel point and the pixel value of the pixel point at the reference coordinate is calculated to obtain a target video frame corresponding to the reference video frame; If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, the pixel value corresponding to the reference coordinate is determined based on the pixel values of the pixel points adjacent to the reference coordinate, and the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate is calculated according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
5. The method according to claim 4, It is characterized in that The determining the pixel value corresponding to the reference coordinate based on the pixel values of the pixel points adjacent to the reference coordinate includes: Based on a nearest neighbor interpolation algorithm, determine the pixel value of the pixel point with the smallest distance from the reference coordinate as the pixel value corresponding to the reference coordinate; or, Based on a bilinear interpolation algorithm, the pixel value corresponding to the reference coordinate is calculated based on the pixel values of a preset number of pixel points within a preset neighborhood range of the reference coordinate.
6. The method according to claim 1, It is characterized in that The adjacent video frames of the reference video frame include: a video frame located before the reference video frame in the original video, and / or a video frame located after the reference video frame.
7. A model training method, It is characterized in that For generating the enhanced parameter determination model according to any one of claims 1 to 6, the method comprises: Acquire a first sample video frame and a second sample video frame; wherein the first sample video frame and the second sample video frame are RAW images; and the first sample video frame is obtained by performing noise addition processing on the second sample video frame; Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, and obtaining the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure; The elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the first sample video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area; For each first image region in the first sample video frame, based on a position of the first image region in the first sample video frame and an offset of the first image region relative to the adjacent video frame, determining a second image region corresponding to the first image region in the adjacent video frame; For each first image region in the first sample video frame, according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, an enhanced pixel value of the first image region is calculated to obtain a predicted video frame corresponding to the first sample video frame; calculating a loss function value representing a difference between the predicted video frame and the second sample video frame; Based on the calculated loss function value, the model parameters of the enhancement parameter determination model of the initial structure are adjusted until a preset convergence condition is reached to obtain a trained enhancement parameter determination model.
8. The method according to claim 7, It is characterized in that The enhanced parameter determination model of the initial structure includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network; The step of inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure to obtain the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure includes: Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, performing feature extraction on the first sample video frame and the adjacent video frames through the first feature extraction network, and obtaining a feature vector representing the image features of the first sample video frame and the image features of the adjacent video frames as a first feature vector; Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector; Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix; Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector; The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
9. A video enhancement device, It is characterized in that The device comprises: A reference video frame determination module, used to determine a designated reference video frame from each video frame of an original video; wherein each video frame in the original video is a RAW image; A matrix acquisition module, used for inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model to obtain an offset matrix and a weight matrix output by the enhancement parameter determination model; The elements in the offset matrix represent the offset of the first image region in the reference video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image region in the reference video frame and the second image region in the adjacent video frame; if an element corresponds to the first image region in the reference video frame, the element represents the weight of the corresponding first image region; if an element corresponds to the second image region in the adjacent video frame, the element represents the weight of the corresponding second image region; an image region determination module, configured to determine, for each first image region in the reference video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the reference video frame and an offset of the first image region relative to the adjacent video frame; a target video frame acquisition module, configured to calculate, for each first image region in the reference video frame, an enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, so as to obtain a target video frame corresponding to the reference video frame; The target video acquisition module is used to obtain the target video corresponding to the original video based on the target video frame.
10. The device according to claim 9, It is characterized in that The enhanced parameter determination model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network; The matrix acquisition module is specifically used for: Inputting the reference video frame and the adjacent video frames of the reference video frame into a pre-trained enhancement parameter determination model, performing feature extraction on the reference video frame and the adjacent video frame through the first feature extraction network, and obtaining a feature vector representing the image features of the reference video frame and the image features of the adjacent video frame as a first feature vector; Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector; Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix; Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector; The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
11. The device according to claim 9, It is characterized in that The image area determination module is specifically used for: For each first image region in the reference video frame, obtaining the coordinates of the first image region in the reference video frame; Calculating a sum of the coordinates of the first image region in the reference video frame and the offset of the first image region relative to the adjacent video frame as a reference coordinate; In the adjacent video frames, an image region corresponding to the reference coordinates is determined to obtain a second image region corresponding to the first image region in the adjacent video frames.
12. The device according to claim 11, It is characterized in that The target video frame acquisition module is specifically used for: For each pixel point in the first image area in the reference video frame, if the reference coordinate corresponding to the pixel point in the adjacent video frame is an integer, according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, a weighted sum of the pixel value of the pixel point and the pixel value of the pixel point at the reference coordinate is calculated to obtain a target video frame corresponding to the reference video frame; If the reference coordinate corresponding to the pixel point in the adjacent video frame is not an integer, the pixel value corresponding to the reference coordinate is determined based on the pixel values of the pixel points adjacent to the reference coordinate, and the weighted sum of the pixel value of the pixel point and the pixel value corresponding to the reference coordinate is calculated according to the weight of the first image area to which the pixel point belongs and the weight of the second image area to which the reference coordinate corresponding to the pixel point in the adjacent video frame belongs, so as to obtain the target video frame corresponding to the reference video frame.
13. The device according to claim 12, It is characterized in that The target video frame acquisition module is specifically used for: Based on a nearest neighbor interpolation algorithm, determine the pixel value of the pixel point with the smallest distance from the reference coordinate as the pixel value corresponding to the reference coordinate; or, Based on a bilinear interpolation algorithm, the pixel value corresponding to the reference coordinate is calculated based on the pixel values of a preset number of pixel points within a preset neighborhood range of the reference coordinate.
14. The device according to claim 9, It is characterized in that The adjacent video frames of the reference video frame include: a video frame located before the reference video frame in the original video, and / or a video frame located after the reference video frame.
15. A model training device, It is characterized in that For generating the enhanced parameter determination model according to any one of claims 1 to 6, the device comprises: A sample video frame acquisition module, used to acquire a first sample video frame and a second sample video frame; wherein the first sample video frame and the second sample video frame are RAW images; and the first sample video frame is obtained by performing noise addition processing on the second sample video frame; A matrix acquisition module, used for inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, and obtaining the offset matrix and the weight matrix output by the enhancement parameter determination model of the initial structure; The elements in the offset matrix represent the offset of the first image area in the first sample video frame relative to the adjacent video frame; the elements in the weight matrix correspond one-to-one to the first image area in the first sample video frame and the second image area in the adjacent video frame; if an element corresponds to the first image area in the first sample video frame, the element represents the weight of the corresponding first image area; if an element corresponds to the second image area in the adjacent video frame, the element represents the weight of the corresponding second image area; an image region determining module, configured to determine, for each first image region in the first sample video frame, a second image region corresponding to the first image region in the adjacent video frame based on a position of the first image region in the first sample video frame and an offset of the first image region relative to the adjacent video frame; a predicted video frame acquisition module, configured to calculate, for each first image region in the first sample video frame, an enhanced pixel value of the first image region according to a weight of the first image region and a weight of a second image region corresponding to the first image region in the adjacent video frame, so as to obtain a predicted video frame corresponding to the first sample video frame; a difference calculation module, configured to calculate a loss function value representing a difference between the predicted video frame and the second sample video frame; The adjustment module is used to adjust the model parameters of the enhancement parameter determination model of the initial structure based on the calculated loss function value until a preset convergence condition is reached to obtain a trained enhancement parameter determination model.
16. The device according to claim 15, It is characterized in that The enhanced parameter determination model of the initial structure includes: a first feature extraction network, a second feature extraction network, a third feature extraction network, an offset prediction network and a weight prediction network; The matrix acquisition module is specifically used for: Inputting the first sample video frame and the adjacent video frames of the first sample video frame in the sample video to which it belongs into the enhancement parameter determination model of the initial structure, performing feature extraction on the first sample video frame and the adjacent video frames through the first feature extraction network, and obtaining a feature vector representing the image features of the first sample video frame and the image features of the adjacent video frames as a first feature vector; Performing feature extraction on the first feature vector by using the second feature extraction network to obtain a second feature vector; Normalizing the second eigenvector by using the offset prediction network to obtain a corresponding offset matrix; Performing feature extraction on the first feature vector by the third feature extraction network to obtain a third feature vector; The third eigenvector is normalized by the weight prediction network to obtain a corresponding weight matrix.
17. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.
18. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the method steps described in any one of claims 7-8 when executing a program stored in a memory.
19. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.
20. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 7-8 are implemented.
Citation Information
Patent Citations
Method for improving video compression coding efficiency based on deep learning
CN107820085A
Video frame reconstruction method
CN111147804A
Image processing method, equipment and medium
CN113763296A
Method and apparatus for video encoding and decoding using matrix-based intra prediction
CN114145021A
Image super-division method and electronic equipment
CN114298900A
