Video image processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a video image processing method and device, electronic equipment and storage medium. The method comprises: performing feature extraction on a current frame image in a to-be-processed video to obtain a first image feature and a first fusion parameter, and obtaining a pixel filter kernel and a filter kernel weight of each pixel in the first image feature; performing pixel filtering and feature fusion on the first image feature based on the pixel filter kernel, the filter kernel weight and the first fusion parameter to obtain a spatial filtering result of the current frame image; if the to-be-processed video comprises a previous frame image adjacent to the current frame image, obtaining a second image feature obtained by performing feature extraction on the previous frame image, and a denoised image obtained by denoising; and obtaining a denoised current frame image based on the second image feature, the denoised image and the spatial filtering result of the current frame image. By using the above method, noise interference in the image can be effectively removed, and the continuity and stability between video image frames can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a video image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, with neural networks becoming a popular solution in machine learning, convolutional neural networks (CNNs) are widely used to handle various image space problems. Numerous denoising methods based on CNNs have been proposed. For example, existing methods use multilayer perceptrons to predict the optimal parameters of a joint bilateral and nonlocal means denoiser for filtering Monte Carlo rendering. Another example is the existing supervised learning framework and deep convolutional neural network kernel prediction model (KPCN), which can obtain more flexible and complex filtering kernels for kernel-based denoising. Yet another example is the existing method that extends the kernel prediction network using an asymmetric loss function, while proposing a multi-resolution denoising structure to utilize information from different scales in image reconstruction. Yet another example is the existing kernel sputtering structure to estimate the contribution of each pixel sample, which can achieve better visual results than kernel aggregation when removing certain specific noise. These methods, based on the idea of ​​spatial filtering, can effectively eliminate noise in a single frame of image.

[0003] However, the inventors discovered through research that when using the current denoising method for video, the detection of each frame is independent of each other, resulting in poor correlation between the denoised frames. This leads to poor continuity and stability between video frames during playback. Summary of the Invention

[0004] In view of this, embodiments of this application propose a video image processing method, apparatus, electronic device, and storage medium, which can utilize a more accurate video image processing model to denoise the video image corresponding to the target user, thereby improving the continuity and stability of the processed video image.

[0005] In a first aspect, embodiments of this application provide a video image processing method, the method comprising: extracting features from a current frame image in a video to be processed to obtain a first image feature of the current frame image and a first fusion parameter corresponding to the first image feature; obtaining a pixel filter kernel and a filter kernel weight corresponding to each pixel based on the pixel features of each pixel included in the first image feature; denoising the pixel features of each pixel in the first image feature using the pixel filter kernel and the filter kernel weight corresponding to the pixel to obtain a denoised pixel feature of each pixel; performing feature fusion on the denoised pixel features of each pixel based on the first fusion parameter to obtain a spatial filtering result of the current frame image; if the video to be processed includes a previous frame image adjacent to the current frame image, obtaining a second image feature obtained by extracting features from the previous frame image, and a denoised previous frame image obtained by denoising the previous frame image; fusing the second image feature, the denoised previous frame image, and the spatial filtering result of the current frame image to obtain a fused image; and denoising the fused image to obtain a denoised current frame image.

[0006] Secondly, embodiments of this application provide a video image processing apparatus, the apparatus comprising: a first data acquisition module, configured to extract features from a current frame image in a video to be processed, obtain a first image feature of the current frame image and a first fusion parameter corresponding to the first image feature, and obtain a pixel filter kernel and a filter kernel weight corresponding to each pixel based on the pixel features of each pixel included in the first image feature; a first filtering module, configured to denoise the pixel features of each pixel in the first image feature using the pixel filter kernel and the filter kernel weight corresponding to the pixel, obtain a denoised pixel feature of each pixel, and perform feature fusion on the denoised pixel features of each pixel based on the first fusion parameter to obtain a spatial filtering result of the current frame image; a second data acquisition module, configured to, when the video to be processed includes a previous frame image adjacent to the current frame image, acquire a second image feature obtained by feature extraction from the previous frame image, and a denoised previous frame image obtained by denoising the previous frame image; and a second filtering module, configured to fuse the second image feature, the denoised previous frame image, and the spatial filtering result of the current frame image to obtain a fused image, and denoise the fused image to obtain a denoised current frame image.

[0007] In one possible implementation, the first data acquisition module is further configured to perform multi-scale feature extraction on the current frame image in the video to be processed, obtaining a first image feature and a first fusion parameter at each feature extraction scale; the first filtering module includes a first denoising processing submodule and a first fusion submodule. The first denoising processing submodule is configured to denoise each pixel feature in the first image features corresponding to each feature extraction scale using the pixel filtering kernel and the filtering kernel weight corresponding to that pixel feature, obtaining the denoised pixel feature corresponding to each pixel; the first fusion submodule is configured to perform feature fusion on the denoised pixel features of each pixel at different feature extraction scales based on the first fusion parameter corresponding to each feature extraction scale, obtaining the spatial filtering result of the current frame image.

[0008] In one possible implementation, the first fusion submodule includes a sampling unit and a fusion unit. The sampling unit is used to downsample the pixel features ranked second in the feature extraction scale from smallest to largest, to obtain a downsampling result, and then upsample the downsampling result to obtain a first upsampled feature. The sampling unit also performs upsampling feature extraction on the pixel features ranked first to obtain a second upsampled feature. The fusion unit is used to perform feature fusion filtering on the pixel features ranked second, based on the second-ranked pixel features, the first upsampled feature, the second upsampled feature, and a first fusion parameter corresponding to the second-ranked pixel features, to obtain the second-ranked pixel features. The sampling unit is further configured to delete the pixel feature ranked first from the sorting order, and then perform downsampling processing on the pixel feature ranked second in the sorting order according to the feature extraction scale from small to large to obtain the downsampling processing result, and perform upsampling processing on the downsampling processing result to obtain the first upsampling feature, and perform upsampling feature extraction on the pixel feature ranked first to obtain the second upsampling feature, and the fusion unit is modified to use the pixel feature ranked second after spatial filtering as the spatial filtering result of the current frame image when the pixel feature ranked second is the last pixel feature in the sorting order.

[0009] In one possible implementation, the fusion unit is further configured to multiply the first upsampled feature with the fusion parameter corresponding to the second-ranked pixel feature to obtain the filtered noise of the second-ranked pixel feature; multiply the second upsampled feature with the fusion parameter corresponding to the second-ranked pixel feature to obtain the canceled noise of the second-ranked pixel feature; subtract the filtered noise from the second-ranked pixel feature and add the canceled noise to obtain the spatially filtered pixel feature of the second-ranked pixel feature.

[0010] In one possible implementation, the first data acquisition module is further configured to perform multi-scale feature extraction on the current frame image in the video to be processed using the trained feature extraction model, and obtain the first image feature of the current frame image at each feature extraction scale and the first fusion parameter corresponding to the first image feature. The trained feature extraction model is trained based on multiple sets of video sample images, and each set of video sample images includes multiple noise sample images belonging to the same video and a denoised sample image corresponding to each noise sample image.

[0011] In one possible implementation, the device further includes: a sample data acquisition module, a spatial filtering module, a temporal filtering module, a loss calculation module, and a model training module.

[0012] The sample feature extraction module is used to input multiple sets of video sample images into the network model, so as to use the network model to extract features from the noise samples in each set of video sample images, and obtain the noise sample features and corresponding sample fusion parameters for each noise sample; based on the sample pixel features included in each pixel of the noise sample features, the sample pixel filtering kernel and sample filtering kernel weights corresponding to each sample pixel are obtained; the spatial filtering module is used to denoise the sample pixel features of each sample pixel in any target noise sample image using the sample pixel filtering kernel and sample filtering kernel weights corresponding to that sample pixel, and obtain the denoised sample pixel features corresponding to each sample pixel; based on the sample fusion parameters, the denoised sample pixel features of each sample pixel are fused to obtain the spatial filtering result of the target noise sample image; the temporal filtering module is used to extract features from the target noise sample image. The video sample image includes a previous frame noise sample image adjacent to the target noise sample image. A denoised previous frame noise sample image is obtained by denoising the previous frame noise sample image. The sample image features of the previous frame noise sample image, the denoised previous frame noise sample image, and the spatial filtering result of the target noise sample image are fused to obtain a sample fusion image. The sample fusion image is then filtered to obtain the denoised result of the target noise sample image. A loss calculation module is used to calculate the model loss based on the denoised result of the target noise sample image, the denoised result of the previous frame noise sample image, the denoised sample image corresponding to the target noise image, and the denoised sample image corresponding to the previous frame noise sample image. A model training module is used to adjust the model parameters of the network model based on the model loss until the model converges, thus obtaining the trained network model.

[0013] In one possible implementation, the loss calculation module includes a spatial loss acquisition submodule, a temporal loss acquisition submodule, and a model loss acquisition submodule. The spatial loss acquisition submodule is used to calculate spatial loss based on the denoising result of the target noise sample image and the corresponding denoised sample image to obtain spatial denoising loss. The temporal loss acquisition submodule is used to calculate loss based on the pixel feature difference between the denoising result of the target noise sample image and the denoising result of the previous frame noise sample image, and the pixel feature difference between the corresponding denoised sample image of the target noise image and the corresponding denoised sample image of the previous frame noise sample image to obtain temporal denoising loss. The model loss acquisition submodule is used to perform a weighted summation of the temporal denoising loss and the spatial denoising loss to obtain model loss.

[0014] In one possible implementation, the first data acquisition module is further configured to map the pixel features of each pixel in the first image features to two feature maps with the same number of channels through convolution, and use the product of the vectors corresponding to the pixel in the two feature maps as the pixel filtering kernel; and use the Softmax function to normalize and calculate the filtering kernel weight corresponding to the pixel filtering kernel.

[0015] In one possible implementation, the second filtering module includes a projection submodule, a stitching submodule, and a fusion submodule. The projection submodule projects the denoised previous frame image onto the current frame image to obtain a projected previous frame image, and projects the second image features onto the current frame image to obtain projected second image features. The stitching submodule performs feature stitching on the projected previous frame image and the projected second image features to obtain a stitched feature map, and performs convolution processing on the stitched features to obtain a temporal feature map. The fusion submodule performs linear fusion on the temporal feature map and the spatial filtering result of the current frame image to obtain a fused image.

[0016] In one possible implementation, the fusion submodule is further configured to perform a weighted summation of the temporal feature map and the spatial filtering result of the current frame image to obtain a fused image, wherein the weight of the temporal feature map is between 0.1 and 0.2.

[0017] In one possible implementation, the second filtering module includes a feature extraction submodule, a parameter acquisition submodule, a second denoising submodule, and a second fusion submodule. The feature extraction submodule is used to extract features from the fused image to obtain fused image features and a second fusion parameter. The parameter acquisition submodule is used to obtain a fused pixel filter kernel and a fused filter kernel weight corresponding to each fused image pixel based on the feature vector included in each fused image pixel in the fused image features. The second denoising submodule is used to denoise each fused pixel feature in the fused image features using the fused pixel filter kernel and the fused filter kernel weight corresponding to that fused pixel feature, obtaining a denoised fused pixel feature corresponding to each pixel. The second fusion submodule is used to perform feature fusion on the denoised fused pixel features of each pixel based on the second fusion parameter to obtain a denoised current frame image.

[0018] In one possible implementation, the first filtering module is further configured to use the spatial filtering result of the current frame image as the denoised current frame image when the previous frame image adjacent to the current frame image is not included in the video to be processed.

[0019] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0021] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device retrieves the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0022] This application provides a video image processing method, apparatus, electronic device, and storage medium. The method includes: extracting features from a current frame image in a video to be processed to obtain a first image feature of the current frame image and a first fusion parameter corresponding to the first image feature; obtaining a pixel filter kernel and a filter kernel weight corresponding to each pixel based on the pixel features of each pixel included in the first image feature; denoising the pixel features of each pixel in the first image feature using the pixel filter kernel and the filter kernel weight corresponding to the pixel to obtain a denoised pixel feature of each pixel; performing feature fusion on the denoised pixel features of each pixel based on the first fusion parameter to obtain a spatial filtering result of the current frame image; if the video to be processed includes a previous frame image adjacent to the current frame image, obtaining a second image feature obtained by extracting features from the previous frame image and a denoised previous frame image obtained by denoising the previous frame image; fusing the second image feature, the denoised previous frame image, and the spatial filtering result of the current frame image to obtain a fused image; and denoising the fused image to obtain a denoised current frame image. By adopting the above method, after obtaining the spatial filtering result, the current frame is denoised based on the denoising result of the previous frame. This enables denoising processing of images in the video in both time and space dimensions, thereby improving the continuity and stability between adjacent images while accurately denoising the video images. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This illustration shows an application scenario diagram of a video image processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating a video image processing method according to an embodiment of this application is shown; Figure 3 This illustration shows a schematic diagram of the structure of a network model provided in an embodiment of this application; Figure 4 A schematic diagram of an image mapping provided in an embodiment of this application is shown; Figure 5 This paper shows another schematic flowchart of a video image processing method proposed in an embodiment of this application; Figure 6 It shows Figure 5 A flowchart illustrating step S240; Figure 7 A projection diagram is shown; Figure 8 It shows Figure 5 A flowchart illustrating step S290; Figure 9 This illustration shows another flowchart of a video image processing method proposed in an embodiment of this application; Figure 10 This application illustrates the current frame image in a video to be processed according to an embodiment of the present application; Figure 11 This illustration shows a schematic diagram of a video image processing method according to an embodiment of this application, which processes the current frame image to obtain the processed current frame image; Figure 12 This illustrates the current frame image in another video to be processed according to an embodiment of this application; Figure 13 It shows Figure 12 Image of region A in the middle; Figure 14 This diagram illustrates the processed region A obtained by processing region A using a video image processing method provided in this application. Figure 15 This diagram illustrates the processing of region A using the SVGF algorithm. Figure 16 This diagram illustrates the processing of region A using the TRMV algorithm. Figure 17 This diagram illustrates the processing of region A using the NTD algorithm. Figure 18 A schematic diagram of the true reference value TD for region A is shown; Figure 19 A comparative diagram of various noise reduction methods used in this application is shown; Figure 20 This diagram shows a connection block diagram of a video image processing apparatus provided in an embodiment of this application; Figure 21 A structural block diagram of an electronic device for performing the methods of embodiments of this application is shown. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0026] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] It should be noted that "multiple" as mentioned in this article refers to two or more.

[0030] With the research and advancement of artificial intelligence technology, it has been studied and applied in multiple fields, playing an increasingly important role.

[0031] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Let's take the application of AI in machine learning as an example: Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. The solution in this application primarily utilizes machine learning for video image processing.

[0032] The following describes an exemplary application of the video image processing method provided by the embodiments of the present invention for performing the above-described video image processing method. The video image processing method provided by the embodiments of the present invention can be applied to, for example... Figure 1 In the server within the application environment shown.

[0033] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application, such as... Figure 1 As shown, this application scenario includes a terminal device 10 and a server 20 that communicates with the terminal device 10 via a network.

[0034] Terminal device 10 may be a mobile phone, computer, tablet computer or vehicle terminal, etc. Terminal device 10 may be equipped with a client for displaying video image processing results, such as a content interaction client, instant messaging client, social networking client or audio and video playback client, etc.

[0035] The network can be a wide area network (WAN), a local area network (LAN), or a combination of both. Terminal device 10 can be a smartphone, smart TV, tablet, laptop, or desktop computer, etc.

[0036] Server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0037] If using such Figure 1The terminal device 10 and server 20 in the process perform video image processing, and the following steps can be specifically executed. Users can upload videos to be processed to server 20 via terminal device 10. When server 20 obtains the video to be processed, it performs feature extraction on the current frame image of the video to be processed, and obtains the first image feature of the current frame image and the first fusion parameter corresponding to the first image feature. Based on the pixel features of each pixel included in the first image feature, it obtains the pixel filtering kernel and filtering kernel weight corresponding to each pixel. For the pixel features of each pixel in the first image feature, it performs denoising using the pixel filtering kernel and filtering kernel weight corresponding to the pixel, and obtains the denoised pixel features of each pixel. Based on the first fusion parameter, it performs feature fusion on the denoised pixel features of each pixel to obtain the spatial filtering result of the current frame image. If the video to be processed includes a previous frame image adjacent to the current frame image, it obtains the second image feature obtained by feature extraction on the previous frame image, and the denoised previous frame image obtained by denoising the previous frame image. It fuses the second image feature, the denoised previous frame image, and the spatial filtering result of the current frame image to obtain a fused image. It denoises the fused image to obtain the denoised current frame image.

[0038] Similarly, based on the above processing steps, each frame of the video to be processed after denoising can be obtained, that is, the denoised video to be processed. After obtaining the denoised video to be processed, the server 20 sends the denoised video to the terminal device 10 so that the terminal device 10 can play the denoised video.

[0039] It should be understood that the above-described method steps can also be executed by the terminal device 10 alone or by the server 20 alone. That is, the application scenarios of the method steps described above are merely illustrative and are not intended to limit this solution.

[0040] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0041] Please read Figure 2 , Figure 2 The present application also provides a video image processing method, which can be applied to an electronic device, such as the aforementioned terminal device 10 or server 20. The method includes: Step S110: Extract features from the current frame image in the video to be processed to obtain the first image feature of the current frame image and the first fusion parameter corresponding to the first image feature. Based on the pixel features of each pixel included in the first image feature, obtain the pixel filter kernel and filter kernel weight corresponding to each pixel.

[0042] The image to be processed includes multiple frames. The current image frame in the video to be processed can be any frame in the image to be processed, such as the first frame, the second frame, or the nth frame, where n is between 1 and N, and N is the number of multiple frames in the video to be processed.

[0043] The process involves feature extraction from images in the video to be processed. Extracted features can include one or more of the following: color features, texture features, shape features, and spatial relationship features. Color features are global features that describe the surface properties of objects within an image or image region; they are pixel-based. Texture features are also global features, describing the surface properties of objects within an image or image region. Unlike color features, texture features are not pixel-based; they require statistical calculations within a region containing multiple pixels. Shape features have two representation methods: contour features and region features. Contour features primarily target the outer boundaries of objects, while region features relate to the entire shape region. Spatial relationship features refer to the spatial positions or relative directions between multiple targets segmented from an image. These relationships can be categorized as connectivity / adjacency, overlap / intersection, and containment / enclosure. Spatial position information is typically divided into two categories: relative spatial position information and absolute spatial position information. The former emphasizes the relative positions between targets, such as up / down and left / right relationships, while the latter emphasizes the distance and orientation between targets.

[0044] Accordingly, when extracting features from the current frame of the image to be processed, one or more of the following algorithms may be used: texture feature extraction, shape feature extraction, and color feature extraction. Texture feature extraction algorithms include methods such as the gray-level co-occurrence matrix (GLCM) method or the Fourier power spectrum method; color feature extraction algorithms include histogram methods, cumulative histogram methods, and color clustering methods. Shape feature extraction algorithms include spatial moment features and other advanced semantic extraction methods such as semantic networks, mathematical logic, and frames. Semantic networks can include convolutional neural networks, recurrent neural networks, deep belief networks, and generative adversarial networks. Therefore, feature extraction from images in a video can be performed using one or more of the following: network models, certain fixed algorithms, and software programs.

[0045] In one possible implementation, image feature information is extracted based on a pre-trained network model. Specifically, when performing feature extraction, the current frame of the video to be detected can be used as input parameters to extract features from the pre-trained network model, thereby obtaining the image features of the current frame.

[0046] The aforementioned pre-trained network model can be a trained convolutional network model, a trained autoregressive model, a trained contrastive learning model, or a trained recurrent neural network model, etc.

[0047] In one possible implementation of this application, a trained convolutional network model can be used to extract features from the current video frame. This model is trained as follows: a sample image set is obtained, comprising sample images, each labeled with feature information. Each sample image in the sample image set is then input into an initial convolutional network model, and the initial convolutional network model is trained based on each sample image to obtain the target network model.

[0048] Specifically, the convolutional neural network mentioned above can be any one of the following: U-net, FPN, FCN, or Segnet. It should be understood that after model training, the fusion parameter corresponding to each feature extraction layer should be a fixed value when extracting features. Furthermore, it should be understood that when there are multiple first image features, these multiple first image features represent image features at multiple resolutions. In this case, the fusion parameters correspond to different resolutions. Moreover, since the fusion parameters are used to fuse image features corresponding to two images with adjacent resolutions, when the number of first image features is n, the number of corresponding first fusion parameters is n-1.

[0049] like Figure 3 As shown, in one possible implementation of this application, the neural network is a U-net network. The U-net network includes a downsampling part, a feature fusion part, and an upsampling part. The downsampling part is a shrinking network that gradually reduces the image size through multiple downsampling layers. During this continuous downsampling process, features can be extracted, and the extracted features are shallow information. Specifically, the input image is processed by two convolutional kernels (e.g., 3x3 convolutions) followed by a max-pooling layer to complete one downsampling operation. For example: an input 572x572 image is processed by two convolutional kernels (3x3) reducing the size from 572 to 570 to 568, and then by a Maxpooling (2x2) layer, reducing the image size to 284. This constitutes one complete downsampling operation, and each subsequent downsampling process follows the same procedure. The downsampling process involves 5 downsampling layers, and the number of image feature channels doubles with each downsampling operation. Figure 3 The number of channels shown ranges from 8 to 258.

[0050] Upsampling networks, also known as dilated networks, can enlarge image sizes through upsampling to extract deeper information. The downsampling process corresponds to the upsampling process, and the number of upsampling layers is the same as the number of downsampling layers. During upsampling, the number of image channels is halved, which is the opposite of the change in the number of feature extraction channels in the left part. Downsampling can be specifically used in the downsampling process.

[0051] In the UNet network, a concatenation network is used to perform concatenation operations. This network consists of multiple concatenation layers, each corresponding to an upsampling layer and a downsampling layer, with each layer performing a single concatenation operation. The purpose of this operation is to obtain fused feature information, combining deep and shallow information. During concatenation, it's crucial that not only are the image sizes consistent (hence cropping to ensure uniformity) but the feature dimensions (channels) are also identical for successful concatenation. It should be understood that the upsampling process incorporates the shallow information extracted during the downsampling process.

[0052] Specifically, in one possible implementation of this application, a lightweight U-Net is used as the backbone network to process the raw input. The features are mapped to new feature spaces at different resolution scales, meaning that the last three decoding layers (upsampling layers) of U-Net output corresponding feature maps (first image features). Additionally, pixel-by-pixel linear blending parameters (first blending parameters) at different resolutions are output separately. The specific output is: ,in, This represents the output of U-Net. and superscript (e.g., 1, 2, 4, etc.) represent multiples relative to the original input resolution. The size of the downsampling is doubled, for each feature map All have a multiple of 8 channels. Specifically, each layer of U-Net uses... convolution of size and using As an activation function, U-Net uses max pooling for downsampling during the encoding stage and simple nearest neighbor interpolation to upsample the results of the previous layer during the decoding stage, thus obtaining the feature extraction results (i.e., feature maps).

[0053] It should be understood that the number of first image features output above is merely illustrative. The number of first image features can also be 2, 4, or 5, etc. Correspondingly, when the number of first image features is 2, the number of first fusion parameters is 1; when the number of first image features is 4, the number of first fusion parameters is 3; and when the number of first image features is 5, the number of first fusion parameters is 4.

[0054] There are several ways to generate pixel filter kernels and filter kernel weights.

[0055] In one possible implementation, the pixel filter kernel and filter kernel weight corresponding to each pixel can be obtained based on the pixel features of each pixel included in the first image features by generating the pixel filter kernel and filter kernel weight for each pixel based on convolution.

[0056] In another possible implementation, the pixel filter kernel and filter kernel weight corresponding to each pixel can be obtained based on the pixel features of each pixel included in the first image features by: mapping the pixel features of each pixel in the first image features to two feature maps with the same number of channels through convolution, taking the product of the vectors corresponding to the pixel in the two feature maps as the pixel filter kernel, and calculating the filter kernel weight corresponding to the pixel filter kernel using a normalization function.

[0057] The normalization function can be of the type Sigmoid function or Softmax function, etc.

[0058] For example, such as Figure 4 As shown, the backbone network U-Net can be used to map the current image frame to the feature space. Above, for each pixel of the current image frame In other words, it stores a vector. Next, through two sizes... The convolution will Mapped to two feature maps, each with 4 channels. and feature map Then, for the current filter center pixel... To generate it and its neighboring pixels between filter kernel weights Simply put corresponding and corresponding The two can be producted:

[0059] In practice, this method can be used to generate pixel filter kernels of size n×n (such as generating...). After obtaining the pixel filter kernel, the weight values ​​can be normalized using the Softmax function to ensure energy conservation. The weight of any pixel coordinate in the neighborhood pixels within the range is ,and , Represented in pixels Centered on, The set of coordinates of neighboring pixels within the range, thus determining the weight values ​​of all filter kernels. And ensured .

[0060] Step S120: For the pixel features of each pixel in the first image features, denoise the pixel features using the pixel filter kernel and filter kernel weight corresponding to the pixel to obtain the denoised pixel features of each pixel. Based on the first fusion parameter, perform feature fusion on the denoised pixel features of each pixel to obtain the spatial filtering result of the current frame image.

[0061] It should be understood that if there are multiple first image features, and these are the image features of the current frame image at multiple resolutions, then denoising can be performed using the pixel filtering kernel and filtering kernel weights corresponding to each pixel at each resolution, resulting in denoised pixel features for each pixel at multiple resolutions. Furthermore, the spatial filtering result of the current frame image can be obtained by fusing the denoised pixel features of each pixel at multiple resolutions based on multiple first fusion parameters.

[0062] The above-mentioned denoising can specifically denoise the image or the shadows in the image, depending on the actual needs.

[0063] Please refer to it again. Figure 3 In one possible implementation, the first image features obtained are three, and are respectively... The first fusion parameter has two values, and they are respectively At that time, and the pixel-wise filtering kernels obtained above at different scale resolutions are respectively The denoising was applied to the noise shadows at the corresponding resolutions, and the resulting three first image features corresponded to the denoised first image features respectively. It should be understood that the pixel features of the pixels in the first image feature after denoising are also denoised. Next, the following formula is used to obtain information at different scales to obtain the final spatial filtering result: ,in, Let U and D represent the denoised shadow images at larger and smaller resolutions, respectively. U represents upsampling by a factor of 2 for nearest neighbor interpolation, and D represents downsampling by a factor of 2 for bilinear interpolation. This yields the spatial filtering results at two adjacent resolutions. By recursively applying the above process to images of adjacent resolutions, the final spatial filtering result can be obtained. .

[0064] Step S130: If the video to be processed includes a previous frame image adjacent to the current frame image, obtain the second image features obtained by feature extraction of the previous frame image, and the denoised previous frame image obtained by denoising the previous frame image.

[0065] In the process of extracting features from the previous frame to obtain the features of the second image, the feature extraction method should be the same as the extraction method for the current frame.

[0066] When extracting features from the previous frame, the process can involve first denoising the previous frame, and then extracting features from the denoised previous frame. In other words, it can involve obtaining the second image features obtained by extracting features from the denoised previous frame. The denoising process for the previous frame should be the same as that for the current frame. It should be understood that when extracting features from the previous frame, features can also be directly extracted from the previous frame adjacent to the current frame in the video being processed.

[0067] The method for extracting the second image features can be the same as the method for extracting the first image features, and the number of second image features can be one or more. When there are multiple second image features, each second image feature has a different resolution.

[0068] In one possible embodiment of this application, the number of second image features is one. Furthermore, the resolution of this second image feature can be the same as the resolution of the previous frame image.

[0069] For example, the previous frame image can be used to extract features using a U-Net network, and the features output by the first upsampling layer or the last downsampling layer in the U-Net network can be used as the second image features.

[0070] Furthermore, it should be understood that the denoising process for the previous frame can be the same as the denoising process for the current frame.

[0071] Step S140: The second image features, the denoised previous frame image, and the spatial filtering result of the current frame image are fused to obtain a fused image, and the fused image is denoised to obtain the denoised current frame image.

[0072] This process involves fusing the second image features, the denoised previous frame image, and the spatial filtering results of the current frame image. Specifically, this can be achieved using a fusion algorithm to obtain a fused image. The fusion algorithm can be a pixel-weighted fusion algorithm, a Laplacian pyramid image fusion algorithm, or an NSST-based image fusion method, depending on the specific requirements.

[0073] After obtaining the fused image, denoising can be performed using an image denoising algorithm or a denoising method similar to step S120. The image denoising algorithm can be a mean filtering algorithm, NL-Means algorithm (Non-Local Means), or BM3D algorithm (Block-matching and 3D filtering), etc., whichever is selected according to the actual needs. They will not be described in detail here.

[0074] Similarly, the same denoising process can be applied to other image frames in the video to be processed, thereby completing the processing of the video.

[0075] It should be understood that, for the first frame of the video to be processed, since it is not affected by other images, there is no need to perform temporal denoising. Accordingly, the previous frame image adjacent to the current frame image is not included in the video to be processed, and the spatial filtering result of the current frame image is used as the denoised current frame image.

[0076] By employing the above method, feature extraction is performed on the current frame image in the video to be processed, obtaining the first image feature of the current frame image and the first fusion parameter corresponding to the first image feature. Based on the pixel features of each pixel included in the first image feature, the pixel filtering kernel and filtering kernel weight corresponding to each pixel are obtained. For the pixel features of each pixel in the first image feature, denoising is performed using the pixel filtering kernel and filtering kernel weight corresponding to that pixel, obtaining the denoised pixel features of each pixel. Based on the first fusion parameter, feature fusion is performed on the denoised pixel features of each pixel to obtain the spatial filtering result of the current frame image. After obtaining the spatial filtering result, the current frame is denoised based on the denoising result of the previous frame image. This can achieve denoising processing of images in both time and space dimensions in the video, thereby improving the continuity and stability between adjacent images while accurately denoising the video images.

[0077] Please see Figure 5 This application provides another video image processing method, which is applied to... Figure 1 The electronic device in the middle, the method includes: Step S210: Perform multi-scale feature extraction on the current frame image in the video to be processed to obtain the first image feature and the first fusion parameter at each feature extraction scale.

[0078] Step S220: Based on the pixel features of each pixel included in the first image features, obtain the pixel filter kernel and filter kernel weight corresponding to each pixel.

[0079] Step S230: For each pixel feature in the first image feature corresponding to each feature extraction scale, denoise the pixel feature using the pixel filter kernel and filter kernel weights corresponding to that pixel feature to obtain the denoised pixel feature for each pixel.

[0080] For a detailed description of steps S210-S240, please refer to the previous description of steps S110-S120.

[0081] In one possible implementation, for each pixel feature in the first image features corresponding to each feature extraction scale, denoising is performed using the pixel filter kernel and filter kernel weights corresponding to that pixel feature, resulting in the denoised pixel feature for each pixel. Specifically, the denoising formula can be... , These are the denoised pixel features in the first image feature set. For the filter kernel weights, These are the pixel features in the first image feature set.

[0082] Among these methods, the filtering approach described above requires fewer parameters and achieves better denoising quality compared to directly using convolutional prediction pixel-by-pixel filtering kernels. Consequently, the denoising efficiency of this approach is also faster.

[0083] Step S240: Based on the first fusion parameter corresponding to each feature extraction scale, perform feature fusion on the denoised pixel features of each pixel at different feature extraction scales to obtain the spatial filtering result of the current frame image.

[0084] For a detailed description of steps S210-S240, please refer to the previous description of steps S110-S120.

[0085] Please see Figure 6 In one optional embodiment of this application, the method of fusing the pixel features of each pixel after denoising at different feature extraction scales based on the first fusion parameter corresponding to each feature extraction scale to obtain the spatial filtering result of the current frame image specifically includes the following steps: Step S241: Downsample the pixel feature ranked second in the feature extraction scale from smallest to largest to obtain the downsampled result, and then upsample the downsampled result to obtain the first upsampled feature. Finally, upsample the pixel feature ranked first to obtain the second upsampled feature.

[0086] Step S242: Based on the second-ranked pixel features, the first upsampled features, the second upsampled features, and the first fusion parameters corresponding to the second-ranked pixel features, perform feature fusion filtering on the second-ranked pixel features to obtain the second-ranked pixel features after spatial filtering.

[0087] In one possible implementation, step S242 may specifically be: multiplying the first upsampled feature with the fusion parameter corresponding to the second-ranked pixel feature to obtain the filtered noise of the second-ranked pixel feature; multiplying the second upsampled feature with the fusion parameter corresponding to the second-ranked pixel feature to obtain the canceled noise of the second-ranked pixel feature; subtracting the filtered noise from the second-ranked pixel feature and adding the canceled noise to obtain the spatially filtered pixel feature of the second-ranked pixel feature.

[0088] Specifically, based on the second-ranked pixel features, the first upsampled feature, the second upsampled feature, and the first fusion parameter corresponding to the second-ranked pixel features, feature fusion filtering is performed on the second-ranked pixel features according to the filtering formula, resulting in spatially filtered pixel features. The filtering formula is as follows: , in the formula, Indicates the first upsampling feature. This indicates the second upsampling feature, where U represents twice the nearest neighbor interpolation upsampling, and D represents twice the bilinear interpolation downsampling. This represents the first fusion parameter corresponding to the first upsampled feature. The pixel features that are spatially filtered after being ranked second are used to achieve inter-layer fusion of pixel features through the above method steps.

[0089] Step S243: Determine whether the second pixel feature in the sorting order is the last pixel feature in the sorting order.

[0090] If so, proceed to step S244: delete the pixel feature ranked first from the sorting order, and return to step S241 until the pixel feature ranked second is the last pixel feature in the sorting order, then proceed to step S245: use the obtained pixel feature ranked second after spatial filtering as the spatial filtering result of the current frame image.

[0091] Step S250: If the video to be processed includes a previous frame image adjacent to the current frame image, obtain the second image features obtained by feature extraction of the previous frame image, and the denoised previous frame image obtained by denoising the previous frame image.

[0092] Step S260: Project the denoised previous frame image onto the current frame image to obtain the projected previous frame image, and project the second image features onto the current frame image to obtain the projected second image features.

[0093] Specifically, the above-mentioned method of projecting the denoised previous frame image and the second image features can be based on traditional motion vectors to project the denoised second image and the second image features; or it can be based on temporal reliable shadow motion vectors to project the denoised second image and the second image features.

[0094] Specifically, if projection is based on traditional motion vectors, then given two consecutive frames, the previous frame... and the current frame Traditional methods can calculate motion vectors representing the pixel correspondence between two frames using back projection. The idea behind back projection is to calculate motion vectors for each pixel intersecting with the main ray. ( (Representing the pixel coordinates in image space) Find its shaded point in the world coordinate system. Pixels corresponding to the previous frame Since the various parameters in the scene rendering process are known, back projection can be accurately calculated without considering optical flow. First, the pixels in the image space... Projected onto the current number In the world coordinate system corresponding to the frame, it is then transformed back to the first frame based on the movement of the scene model geometry. In the world coordinate system of the frame, the transformed world coordinates are finally in the [frame name missing]. If the image is projected back into image space from the frame, then the target pixel is obtained at this point. The above process is represented by formulas below. Definition Let be the projection matrix, representing the projection matrix in each frame. Transformation (the transformation from screen space to model space for each frame). Let be the transformation matrix of the object's geometric spatial position between each two frames. Then, the formula for calculating the back projection can be written as: The corresponding motion vector can be easily obtained from this formula, i.e. It is defined as the difference between the pixel coordinates of the previous frame obtained by back projection in the image space and the pixel coordinates of the current frame.

[0095] Specifically, if projection is based on a temporally reliable shadow motion vector, a more accurate temporal correspondence between pixels can be found because the temporally reliable motion vector can be used. In one embodiment of this application, to achieve noise reduction in the shadow area, the movement of the shadow is tracked based on the changing positions of the occluder and the light source over time, such as... Figure 7 As shown. In the current frame i, for each pixel in shadow... (Here, the superscript 'v' indicates visibility, which is the coordinate of a received shadow. During rendering, a 1spp downsampled shadow image is obtained, where each pixel has a value of 0 or 1. 0 indicates that the pixel is in shadow, and 1ssp means that the pixel sample is 1.) The projection matrix P can be used to accurately determine the coordinates of the corresponding shading point in world space. Coordinates of the obstruction and the coordinates of the light source sampling points Simultaneously, the geometric transformation matrix between the occluder and the light source sampling point can be quickly obtained in two consecutive frames, thereby calculating the coordinates of the occluder in world space in the previous frame. and the coordinates of the light source sampling point Here we make an assumption: if the shading point receives the shadow... If the surrounding geometry is planar, then the plane and its connection can be found in the previous frame. and The intersection point between the lines in world space corresponds to the shadow shading point in the previous frame. Finally, this intersection point is projected onto the image space in the previous frame, and its corresponding pixel is... That is what we need to get from The tracking coordinates of the shadow in the previous frame image. The formula is expressed as follows: in, The function represents a ray emitted from point x to point y. Let x represent the plane containing the point x, and the function... Represents rays The intersection point with the y-plane. For different light source sampling points, occluders, and shadow shading points, the transformation matrix... There may be differences. Using the above formula, a more reliable shadow motion vector can be obtained, i.e. .

[0096] Wherein, if the shadow motion vector can be used This indicates that, correspondingly, the previous frame image after denoising. The second image feature is Using temporally reliable shadow motion vectors The projected image of the previous frame obtained from the projection: The features of the second image after projection are .

[0097] Step S270: Perform feature concatenation on the features of the previous frame image after projection and the second image after projection to obtain a concatenated feature map, and perform convolution on the concatenated features to obtain a temporal feature map.

[0098] The convolutional processing of the concatenated feature maps can be performed using n×n convolutions, either once or multiple times. For example, a temporal feature map can be generated using two 3×3 convolutional layers.

[0099] Step S280: Linearly fuse the temporal feature map with the spatial filtering result of the current frame image to obtain the fused image.

[0100] Specifically, step S280 can involve weighted summation of the temporal feature map and the spatial filtering result of the current frame image to obtain a fused image. The weights of the temporal feature map are between 0.1 and 0.2.

[0101] In other words, the following linear fusion formula can be used to obtain the fused image when performing linear fusion: In the formula, Represents the time-domain feature map. This represents the spatial filtering result of the current frame image. The spatial filtering result of each frame is first stored in... Then, compare the pixel values ​​with those of the previous frame. Perform linear mixing. It should be noted that... It is a scalar between 0 and 1, which determines the weights used when applying time-domain information. Typically, in practical applications... The value is limited to between 0.1 and 0.2, indicating that this type of method is highly dependent on the temporal information of the previous frame. As more and more frames are rendered, the temporal filtering process continues, and the denoised image gradually accumulates to a result that is closer to the true value. As can be seen from the linear fusion formula above, the contribution of each frame's value to the result decreases exponentially over time.

[0102] It should be noted that this was done to address issues caused by non-planar shadow receivers. After calculating the back projection... Then, the algorithm will measure the normal to the assumed shadow-receiving plane. With direction The angle between This is used to adjust the linear blending formula used in the time filtering of shadows. In one possible implementation, a neural network can also be used to infer the "optimal" linear mixing parameters. .

[0103] Step S290: Denoise the fused image to obtain the denoised current frame image.

[0104] For a detailed description of denoising the fused image to obtain the denoised current frame image, please refer to the previous description of step S140. That is, denoising the fused image can be performed using an image denoising algorithm or a denoising method similar to step S120.

[0105] Please see Figure 8 In one possible implementation, step S290 includes: Step S291: Extract features from the fused image to obtain fused image features and second fusion parameters.

[0106] Step S292: Based on the feature vector included in each fused image pixel in the fused image features, obtain the fused pixel filter kernel and fused filter kernel weight corresponding to each fused image pixel.

[0107] Step S293: For each fused pixel feature in the fused image features, denoise it using the fused pixel filter kernel and the weight of the fused filter kernel corresponding to the fused pixel feature, and obtain the denoised fused pixel feature corresponding to each pixel.

[0108] Step S294: Based on the second fusion parameter, perform feature fusion on the fused pixel features of each pixel after denoising to obtain the denoised current frame image.

[0109] For a detailed description of steps S291-S294 above, please refer to the detailed description of steps S110 and S120 above, as well as the detailed description of steps S210-S240 above. They will not be repeated here.

[0110] By employing the above method, multi-scale feature extraction is performed on the current frame image of the video to be processed, obtaining the first image feature and the first fusion parameter at each feature extraction scale. For each pixel feature in the first image feature corresponding to each feature extraction scale, denoising is performed using the pixel filtering kernel and filtering kernel weight corresponding to that pixel feature, resulting in the denoised pixel feature for each pixel. Based on the first fusion parameter corresponding to each feature extraction scale, feature fusion is performed on the denoised pixel features of each pixel at different feature extraction scales to obtain the spatial filtering result of the current frame image. This achieves accurate spatial filtering results based on multi-scale feature extraction results. After obtaining the spatial filtering results, the current frame is denoised based on the denoising result of the previous frame image. This enables denoising processing of images in both time and space dimensions in the video, thereby improving the continuity and stability between adjacent images while accurately denoising the video images. Furthermore, by employing multi-scale feature extraction, this method can be applied to real-time denoising of dynamic shadows in complex scenes. Additionally, when projecting using temporally reliable shadow running vectors, this application is suitable for large-area light scenes with high-speed movement, effectively eliminating soft shadow noise while acquiring more accurate shadow details, and effectively solving the ghosting problem caused by rapid movement of light sources, viewpoints, and objects. Moreover, using the methods described above, high-quality reconstruction results can be recovered from high-frequency shadow noise generated by ray tracing with a small number of samples per pixel, avoiding problems such as excessive blurring and loss of detail.

[0111] Please see Figure 9 This application provides yet another video image processing method, which is applied to... Figure 1 The electronic device in the middle, the method includes: Step S310: Input multiple sets of video sample images into the network model respectively, so as to use the network model to extract features from the noise samples in each set of video sample images, and obtain the noise sample features and corresponding sample fusion parameters of each noise sample.

[0112] Among them, multiple sets of video samples can be multiple animation sequence sample images generated in multiple scenes.

[0113] For example, 160 animation sequences, each 8 frames long, were generated across 5 different scenes, totaling 1280 frames of data with a resolution of 1280×720. For each frame, the original shadow noise map v was generated using ray tracing at 1 spp, while the ground truth reference value r was rendered at 2048 spp to ensure convergence. Simultaneously, each scene utilized a large, fast-moving, square light source to ensure sufficiently complex and dynamically changing shadows.

[0114] Specifically, the network model described above can be any one of U-net network, FPN network, FCN network, or Segnet network. In this application, the network model described above is a U-net network.

[0115] By utilizing the above model, the denoising process of video images can be described as a process of obtaining an image from a low-sampled image. To the true reference value under high sampling Mapping between .generally, It is related to the coloring point Related tuples, such as .in This represents the noise value under low sampling, which in this invention is represented as the shadow obtained at 1 spp. Its value is . This represents auxiliary features, such as surface normals or depth (here, depth represents the distance between the viewpoint and the shading point). When using deep learning, for mapping... The solution can be expressed as a set of representatives Neural network parameter weights Training. Through supervised learning, utilizing... Dataset of sample pairs Through the loss function Optimization yields the best estimated parameters : .in, express The set of vectors of pixels in the neighborhood, used to... Reconstruct a reasonable output.

[0116] For example, if it is necessary to filter out noise from shadows in a video image, the number of channels in the video sample image input to the feature network can be 8. The specific number of channels is determined by the number of shadows generated by ray tracing at 1 spp. Surface normals with 3 channels Depth of 1 channel design (wherein, depth) This represents the distance between the viewpoint and the shading point, and two channels of motion vectors (such as the shadow motion vector in the above embodiment). And the ratio of the distance from the shadow receiving point of a channel to the occluder to the distance from the occluder to the sampling light source. Among them, the original shadow Since visibility information is represented by a value of 0 for shadowed areas and 1 for visible areas, to improve the network's response to shadowed areas, [the following is used:] Set the shadow value to 1 to obtain a new shadow map. Replace the original shadow Input the network. For depth... Because the spatial scale of each scene model is different, in order to ensure that the network learns geometric relationships rather than specific distances, the depth is... Normalization was performed to adjust its size. Similarly, Similar normalization operations were also performed.

[0117] Step S320: Based on the sample pixel features included in each pixel in the noise sample features, obtain the sample pixel filter kernel and sample filter kernel weight corresponding to each sample pixel.

[0118] Step S330: For the sample pixel features of each sample pixel in any target noise sample image, denoise by using the sample pixel filtering kernel and sample filtering kernel weight corresponding to the sample pixel to obtain the denoised sample pixel features corresponding to each sample pixel. Based on the sample fusion parameters, feature fusion is performed on the denoised sample pixel features of each sample pixel to obtain the spatial filtering result of the target noise sample image.

[0119] Step S340: If the video sample image to which the target noise sample image belongs includes a previous frame noise sample image adjacent to the target noise sample image, obtain the denoised previous frame noise sample image obtained by denoising the previous frame noise sample image.

[0120] Step S350: The sample image features of the previous frame noise sample image, the denoised previous frame sample image, and the spatial filtering result of the target noise sample image are fused to obtain a sample fusion image. The sample fusion image is then filtered to obtain the denoising result of the target noise sample image.

[0121] For a detailed description of steps S310-S340 above, please refer to the detailed description of steps S110-S140 above.

[0122] Step S360: Calculate the loss based on the denoising result of the target noise sample image, the denoising result of the previous frame noise sample image, the denoised sample image corresponding to the target noise image, and the denoised sample image corresponding to the previous frame noise sample image to obtain the model loss.

[0123] In one possible implementation, step S360 may be: performing spatial loss calculation based on the denoising result of the target noise sample image and the denoised sample image corresponding to the target noise image to obtain spatial denoising loss; performing loss calculation based on the pixel feature difference between the denoising result of the target noise sample image and the denoising result of the previous frame noise sample image, and the pixel feature difference between the denoised sample image corresponding to the target noise image and the denoised sample image corresponding to the previous frame noise sample image to obtain temporal denoising loss; and calculating a model loss by weighting the spatial denoising loss and the temporal denoising loss, calculating the average, or finding the maximum value.

[0124] In step S160 above, the spatial denoising loss can specifically be the sum of the current denoising result and the true value. Loss function calculation: Temporal denoising loss is expressed as the difference between corresponding pixels between two consecutive frames and uses... The loss function is calculated. Adding the spatial loss to the temporal loss yields the complete loss function. ,in, These represent the network denoising result and the true reference value, respectively. These represent the current frame and the previous frame, respectively. Furthermore, for each consecutive frame segment, only the spatial portion is calculated for the first frame; the full loss function is used for backpropagation iterations only for subsequent frames. The L1 loss function mentioned above can be either the cross-entropy loss function or the mean squared error loss function, depending on the actual requirements.

[0125] Step S370: Adjust the model parameters of the network model based on the model loss until the model converges, and obtain the trained network model.

[0126] During network model training, the Adam optimizer can be used with an initial learning rate of 0.001, while other parameters remain at their default values. Model convergence is confirmed when the model has completed a preset number of training iterations, such as 1024 iterations. It's important to note that the learning rate decreases by 50% every 128 iterations, the batch size is 8 bytes per iteration, and during training, 512×512 local blocks are randomly cropped from the original image size and randomly flipped horizontally, vertically, and horizontally to expand the dataset. The model can be executed on an electronic device; for example, training can be performed on an electronic device equipped with an RTX 3090 graphics card and an Intel Core i9-10900K processor.

[0127] By employing the steps S310-S370 described above, the denoising process of this application can be described as a process of obtaining an image from a low-sampled image. To the true reference value under high sampling Mapping between .in, It is related to the coloring point Related tuples, such as .in This represents the noise value under low sampling, specifically the shadow obtained at 1 spp. Its value is . This represents auxiliary features, such as surface normals or depth (here, depth represents the distance between the viewpoint and the shading point). When using deep learning, for mapping... The solution can be expressed as a set of representatives Neural network parameter weights Training. Through supervised learning, utilizing... Dataset of sample pairs Through the loss function Optimization yields the best estimated parameters : In the formula, express The set of vectors of pixels in the neighborhood, used to... Reconstruct a reasonable output.

[0128] After obtaining the network model described above, feature extraction can be performed using the network model, and based on the corresponding steps in the video image processing methods described in the two embodiments above, noise reduction can be applied to the video to be processed. Figure 10 and Figure 11 As shown, Figure 10 The current frame image in the video to be processed. Figure 11 The current frame image is the denoised version obtained by performing feature extraction using the network model of this application and then executing the video processing method described above. The image in the video to be processed can be a shadow noise image generated by ray tracing at a low sampling rate. After processing the image using the processing method described above, high-quality shadow results can be produced, improving the visual performance of the image. Furthermore, it should be noted that the aforementioned network model can be integrated into the post-processing module of an existing rendering pipeline for image processing.

[0129] To more accurately describe this invention, a comparative analysis will be conducted below of the video image processing method provided by this invention and two non-network denoising methods, SVGF and TRMV, as well as a network denoising method, NTD. NTD uses a small network model, and furthermore, to maintain a similar parameter size and runtime as the network of this invention, it is designed to work in a similar manner. Furthermore, since this application can denoise shadows, color-related albedo is removed from the network input of NTD.

[0130] like Figure 12-15 As shown, to more clearly compare the differences between the solution provided in this application and the prior art, the method of this application will be used to denoise region A of the current frame image in the video image. The image cropped from region A is shown below. Figure 13 As shown, Figure 14-18 The denoising results of this application (Ours), SVGF algorithm, TRMV algorithm, and NTD algorithm in the test scene are shown in turn. It can be seen that the method of this application, under severe noise input in a high-speed moving scene with a large area light source, reconstructs an image that is relatively close to the true reference value (GT), effectively eliminating soft shadow noise while obtaining more accurate shadow details. Other solutions have some obvious shortcomings. SVGF adjusts the shape and size of the spatial filter by analyzing the spatial and temporal variance of the image; however, it uses traditional motion vectors to obtain temporal information, which is completely ineffective in the data designed in this invention. Because in the scene data, apart from the high-speed movement of the light source, the viewpoint and object geometry remain stationary, meaning that the traditional motion vectors tracking object geometry will all be 0, thus failing to correctly find the corresponding shadow in the previous frame. Due to the use of incorrect temporal information, the denoising results of SVGF show obvious shadow lag (ghosting) and excessive blurring of shadow details. TRMV uses temporally reliable shadow motion vectors, resulting in good denoising results. However, due to the limitations of its denoising filter kernel, it cannot handle complex situations, thus reconstructing less shadow detail. NTD uses the network to directly predict pixel-wise filter kernels for denoising. Although its recurrent feedback uses traditional motion vectors for reprojection, resulting in inaccurate temporal information, it achieves very good denoising results using only spatial filtering, thanks to the powerful modeling capabilities of neural networks. However, compared to the results of this invention, the image reconstructed by NTD still suffers from problems such as loss of some soft shadow details and poor preservation of hard shadows and object geometric boundaries.

[0131] It should be noted that when comparing this application with existing non-network denoising methods and network denoising methods, the peak signal-to-noise ratio (PSNR) can be used to evaluate the denoising quality of a single frame. In the formula, MSE represents the mean square error between corresponding pixels in two images. PSNR is calculated based on the error between corresponding pixels in two images; a higher PSNR indicates less image distortion. Table 1 shows the results of evaluating single-frame denoising quality using PSNR.

[0132] Beyond improving the denoising quality of individual frames, another major contribution is the use of temporally reliable shadow motion vectors combined with a temporal fusion module, which provides better inter-frame stability. To evaluate this, this application uses temporal peak signal-to-noise ratio (tPSNR) to calculate the inter-frame stability across consecutive frames. Specifically, it calculates the pixel difference between the current frame and the previous frame. , Table 2 shows the results of evaluating inter-frame coherence using tPSNR.

[0133] Furthermore, the goal of this application is real-time shadow denoising, so the post-processing runtime is also a very important performance indicator. Therefore, Table 3 shows the runtime of the above-mentioned schemes. All tests are for 1280×720 resolution images, and the hardware devices are kept consistent, using RTX 3090 and Intel Core i9-10900K. In Falcor

[21] , ray tracing using 1spp takes about 15ms per frame in the most complex scene (bistro exterior) of this invention. For real-time applications, at least 30 frames per second should be able to be displayed, that is, the time to process one frame should preferably not exceed 30ms. Table 3 is as follows:

[0134] As shown in Table 3, the method used in this application takes an average of approximately 13.2 ms, which is sufficient for real-time performance. Although it is still slower than methods such as SVGF and TRMV that do not use neural networks, it produces good denoising results.

[0135] Furthermore, to further clarify the effects of this application, firstly as follows... Figure 19The image corresponding to row (a) shows the result (Ours) obtained by denoising the current image frame (Input) using the direct prediction pixel-wise filtering kernel, and the comparison result of the denoising by the kernel prediction module (Ours-k) used to perform steps S220-S230 with the true reference value (GT). It can be seen that the direct prediction pixel-wise filtering kernel still has relatively obvious noise at the boundary between hard shadows and soft shadows, and cannot transition in a smooth way, while also losing some details in the soft shadow area. In addition to the loss in quality, the direct prediction pixel-wise filtering kernel method also brings a larger number of network parameters. Taking the kernel prediction module (Ours-k) as an example, to generate a 5×5 pixel-wise filtering kernel of size from an 8-channel tensor output, i.e., a 25-channel tensor, the direct prediction method requires 25 convolutional kernels of size 1×1×8, while the kernel prediction module (Ours-k) only needs 8 convolutional kernels. Furthermore, the kernel prediction module (Ours-k) generates the filtering kernel weights based on the dot product of the neighborhood feature vectors, which means that filtering kernels of any size can be flexibly generated.

[0136] Next, as Figure 19 In the image corresponding to row (b) in the diagram, to verify the importance of using accurate temporal information, the traditional motion vector (Ours) and the shadow motion vector (Ours-m) used for the current image frame (Input) are compared with the true reference value (GT). It is easy to see that the denoising result produces obvious white spots in the large soft shadow areas with fast motion, which is precisely due to noise caused by inaccurate temporal reprojection.

[0137] Finally, as Figure 19 In the image corresponding to (c), the ratio of the distance from the receiving point to the occluder to the distance from the occluder to the sampling light source for the current image frame (Input) before removing shadows from the network input is shown. (Ours) and the ratio of the distance from the shadow receiver point to the occluder to the distance from the occluder to the sampling light source in the network input. The results of denoising using the network (Ours-r) after training are compared with the true reference value (GT). This input feature reveals, to some extent, the hardness or softness of the shadow; that is, the smaller the ratio, the harder the shadow should be, and vice versa. It can be seen that the denoising was removed... Subsequently, the network could no longer accurately reconstruct the details of hard shadows within large areas of soft shadows, resulting in a more blurred overall visual effect.

[0138] In addition, Tables 1 and 2 respectively show the quantitative evaluation results of the ablation experiment, verifying the effectiveness of the kernel prediction module, shadow motion vector and occlusion-light source distance ratio features involved in the present invention for the experimental results.

[0139] Based on the above analysis, it can be seen that for the multi-scale hierarchical denoising network for dynamic shadows, after using the kernel prediction module, a temporal reliable shadow motion vector is introduced, thereby making more effective use of continuous frame information and achieving high-quality denoising of real-time dynamic shadows. Experimental results show that this application is suitable for real-time denoising of dynamic shadows in complex scenes, especially in large-area light scenes with high-speed movement. It can effectively eliminate soft shadow noise while obtaining more accurate shadow details, and effectively solves problems such as shadow ghosting when light sources, viewpoints, and objects move quickly.

[0140] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0141] Figure 20 The video image processing apparatus 400 shown in one embodiment is as follows: Figure 20 As shown, the video image processing device 400 includes: a first data acquisition module 410, a first filtering module 420, a second data acquisition module 430, and a second filtering module 440.

[0142] The first data acquisition module 410 is used to extract features from the current frame image in the video to be processed, obtain the first image features of the current frame image and the first fusion parameter corresponding to the first image features, and obtain the pixel filter kernel and filter kernel weight corresponding to each pixel based on the pixel features of each pixel included in the first image features; the first filtering module 420 is used to denoise the pixel features of each pixel in the first image features using the pixel filter kernel and filter kernel weight corresponding to the pixel, obtain the denoised pixel features of each pixel, and perform feature fusion on the denoised pixel features of each pixel based on the first fusion parameter to obtain the spatial filtering result of the current frame image; the second data acquisition module 430 is used to acquire the second image features obtained by feature extraction on the previous frame image and the denoised previous frame image obtained by denoising the previous frame image when the video to be processed includes the previous frame image adjacent to the current frame image; the second filtering module 440 is used to fuse the second image features, the denoised previous frame image and the spatial filtering result of the current frame image to obtain a fused image, and denoise the fused image to obtain the denoised current frame image.

[0143] In one possible implementation, the first data acquisition module 410 is further configured to perform multi-scale feature extraction on the current frame image in the video to be processed, obtaining a first image feature and a first fusion parameter at each feature extraction scale; the first filtering module includes a first denoising processing submodule and a first fusion submodule. The first denoising processing submodule is configured to denoise each pixel feature in the first image features corresponding to each feature extraction scale using the pixel filtering kernel and the filtering kernel weight corresponding to that pixel feature, obtaining the denoised pixel feature corresponding to each pixel; the first fusion submodule is configured to perform feature fusion on the denoised pixel features of each pixel at different feature extraction scales based on the first fusion parameter corresponding to each feature extraction scale, obtaining the spatial filtering result of the current frame image.

[0144] In one possible implementation, the first fusion submodule includes a sampling unit and a fusion unit. The sampling unit is used to downsample the pixel features ranked second in the feature extraction scale from smallest to largest, to obtain a downsampling result, and then upsample the downsampling result to obtain a first upsampled feature. The sampling unit also performs upsampling feature extraction on the pixel features ranked first to obtain a second upsampled feature. The fusion unit is used to perform feature fusion filtering on the pixel features ranked second, based on the second-ranked pixel features, the first upsampled feature, the second upsampled feature, and a first fusion parameter corresponding to the second-ranked pixel features, to obtain the second-ranked pixel features. The pixel features are spatially filtered after the pixel features are spatially filtered; the sampling unit is also used to delete the pixel feature ranked first from the sorting order, and to downsample the pixel feature ranked second in the sorting order of the feature extraction scale from small to large to obtain the downsampling result, and to upsample the downsampling result to obtain the first upsampled feature, and to extract the second upsampled feature from the pixel feature ranked first. The fusion unit is used to use the pixel feature ranked second after spatial filtering as the spatial filtering result of the current frame image when the pixel feature ranked second is the last pixel feature in the sorting order.

[0145] In one possible implementation, the fusion unit is further configured to multiply the first upsampled feature with the fusion parameters corresponding to the second-ranked pixel features to obtain the filtered noise of the second-ranked pixel features; multiply the second upsampled feature with the fusion parameters corresponding to the second-ranked pixel features to obtain the canceled noise of the second-ranked pixel features; subtract the filtered noise from the second-ranked pixel features and add the canceled noise to obtain the spatially filtered pixel features of the second-ranked pixel features.

[0146] In one possible implementation, the first data acquisition module 410 is further configured to use the trained feature extraction model to perform multi-scale feature extraction on the current frame image in the video to be processed, and obtain the first image feature of the current frame image at each feature extraction scale and the first fusion parameter corresponding to the first image feature. The trained feature extraction model is trained based on multiple sets of video sample images, and each set of video sample images includes multiple noise sample images belonging to the same video and a denoised sample image corresponding to each noise sample image.

[0147] In one possible implementation, the first filtering module 420 is further configured to use the spatial filtering result of the current frame image as the denoised current frame image when the previous frame image adjacent to the current frame image is not included in the video to be processed.

[0148] In one possible implementation, the first data acquisition module 410 is further configured to map the pixel features of each pixel in the first image features to two feature maps with the same number of channels through convolution, and use the product of the vectors corresponding to the pixel in the two feature maps as the pixel filtering kernel corresponding to the pixel; and use the Softmax function to normalize and calculate the filtering kernel weight corresponding to the pixel filtering kernel.

[0149] In one possible implementation, the second filtering module 440 includes a projection submodule, a stitching submodule, and a fusion submodule. The projection submodule projects the denoised previous frame image onto the current frame image to obtain a projected previous frame image, and projects the second image features onto the current frame image to obtain projected second image features. The stitching submodule performs feature stitching on the projected previous frame image and the projected second image features to obtain a stitched feature map, and performs convolution processing on the stitched features to obtain a temporal feature map. The fusion submodule performs linear fusion on the temporal feature map and the spatial filtering result of the current frame image to obtain a fused image.

[0150] In one possible implementation, the fusion submodule is further configured to perform a weighted summation of the temporal feature map and the spatial filtering result of the current frame image to obtain a fused image, wherein the weight of the temporal feature map is between 0.1 and 0.2.

[0151] In one possible implementation, the second filtering module 440 includes a feature extraction submodule, a parameter acquisition submodule, a second denoising submodule, and a second fusion submodule. The feature extraction submodule is used to extract features from the fused image to obtain fused image features and a second fusion parameter. The parameter acquisition submodule is used to obtain a fused pixel filter kernel and fused filter kernel weights corresponding to each fused image pixel based on the feature vector included in each fused image pixel in the fused image features. The second denoising submodule is used to denoise each fused pixel feature in the fused image features using the fused pixel filter kernel and fused filter kernel weights corresponding to that fused pixel feature, obtaining a denoised fused pixel feature corresponding to each pixel. The second fusion submodule is used to perform feature fusion on the denoised fused pixel features of each pixel based on the second fusion parameter to obtain a denoised current frame image.

[0152] In one possible implementation, the video image processing device 400 further includes: a sample data acquisition module, a spatial filtering module, a temporal filtering module, a loss calculation module, and a model training module.

[0153] The sample feature extraction module is used to input multiple sets of video sample images into the network model, so that the network model can extract features from the noise samples in each set of video sample images, and obtain the noise sample features and corresponding sample fusion parameters for each noise sample. Based on the sample pixel features included in each pixel of the noise sample features, the sample pixel filtering kernel and sample filtering kernel weights corresponding to each sample pixel are obtained. The spatial filtering module is used to denoise the sample pixel features of each sample pixel in any target noise sample image using the sample pixel filtering kernel and sample filtering kernel weights corresponding to that sample pixel, and obtain the denoised sample pixel features corresponding to each sample pixel. Based on the sample fusion parameters, the denoised sample pixel features of each sample pixel are fused to obtain the spatial filtering result of the target noise sample image. The temporal filtering module is used to filter the target noise sample image. The video sample image to which the image belongs includes the previous frame noise sample image adjacent to the target noise sample image. A denoised previous frame noise sample image is obtained by denoising the previous frame noise sample image. The sample image features of the previous frame noise sample image, the denoised previous frame noise sample image, and the spatial filtering result of the target noise sample image are fused to obtain a sample fusion image. The sample fusion image is then filtered to obtain the denoised result of the target noise sample image. A loss calculation module is used to calculate the model loss based on the denoised result of the target noise sample image, the denoised result of the previous frame noise sample image, the denoised sample image corresponding to the target noise image, and the denoised sample image corresponding to the previous frame noise sample image. A model training module is used to adjust the model parameters of the network model based on the model loss until the model converges, obtaining the trained network model.

[0154] In one possible implementation, the loss calculation module includes a spatial loss acquisition submodule, a temporal loss acquisition submodule, and a model loss acquisition submodule. The spatial loss acquisition submodule is used to calculate spatial loss based on the denoising result of the target noise sample image and the corresponding denoised sample image to obtain spatial denoising loss. The temporal loss acquisition submodule is used to calculate loss based on the pixel feature difference between the denoising result of the target noise sample image and the denoising result of the previous frame noise sample image, and the pixel feature difference between the denoised sample image corresponding to the target noise image and the corresponding denoised sample image of the previous frame noise sample image to obtain temporal denoising loss. The model loss acquisition submodule is used to perform a weighted summation of the temporal denoising loss and the spatial denoising loss to obtain the model loss.

[0155] The following will combine Figure 21 This application describes an electronic device.

[0156] Please see Figure 21 Based on the video image processing model training method provided in the above embodiments, this application embodiment also provides another electronic device 100 including a processor 102 capable of executing the aforementioned method. The electronic device 100 can be a server or a terminal device, and the terminal device can be a smartphone, tablet computer, computer, or portable computer, etc.

[0157] The electronic device 100 also includes a memory 104. The memory 104 stores a program that can execute the contents of the foregoing embodiments, and the processor 102 can execute the program stored in the memory 104.

[0158] The processor 102 may include one or more cores for data processing and message matrix units. The processor 102 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 102 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102 and may be implemented separately using a communication chip.

[0159] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data acquired by the electronic device 100 during use (e.g., video to be processed, first image features, second image features, and spatial filtering results).

[0160] The electronic device 100 may also include a network module and a screen. The network module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices, such as audio playback devices. The network module may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen can display interface content and perform data interaction, such as displaying the video image processing results of the video to be processed.

[0161] In some embodiments, the electronic device 100 may further include a peripheral interface 106 and at least one peripheral device. The processor 102, memory 104, and peripheral interface 106 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency component 108, a positioning component 112, a camera 114, an audio component 116, a display screen 118, and a power supply 122. Peripheral interface 106 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 102 and memory 104. In some embodiments, processor 102, memory 104 and peripheral interface 106 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 102, memory 104 and peripheral interface 106 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0162] The radio frequency (RF) component 108 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF component 108 communicates with communication networks and other communication devices via electromagnetic signals. The RF component 108 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF component 108 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF component 108 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF component 108 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0163] Positioning component 112 is used to locate the current geographic location of an electronic device to enable navigation or LBS (Location Based Service). Positioning component 112 can be a positioning component based on the US GPS (Global Positioning System), BeiDou system, or Galileo system.

[0164] Camera 114 is used to capture images or videos. Optionally, camera 114 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device 100, and the rear-facing camera is located on the back of the electronic device 100. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera 114 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0165] Audio component 116 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and environment, converting them into electrical signals that are input to processor 102 for processing, or input to radio frequency component 108 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the electronic device 100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 102 or radio frequency component 108 into sound waves. The speaker may be a conventional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, audio component 116 may also include a headphone jack.

[0166] Display screen 118 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 118 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 102 for processing. In this case, display screen 118 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 118, which serves as the front panel of electronic device 100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of electronic device 100 or in a folded design; in still other embodiments, display screen 118 may be a flexible display screen, disposed on a curved or folded surface of electronic device 100. Furthermore, display screen 118 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 118 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0167] Power supply 122 is used to supply power to various components in electronic device 100. Power supply 122 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 122 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0168] This application also provides a structural block diagram of a computer-readable storage medium. The computer-readable medium stores program code, which can be called by a processor to execute the methods described in the above method embodiments.

[0169] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0170] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various optional implementations above.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video image processing method, characterized in that, The method includes: Feature extraction is performed on the current frame image in the video to be processed to obtain the first image feature of the current frame image and the first fusion parameter corresponding to the first image feature. The pixel features of each pixel in the first image feature are mapped to two feature maps with the same number of channels through convolution. The product of the vectors corresponding to the pixel in the two feature maps is used as the pixel filtering kernel corresponding to the pixel. The filtering kernel weight corresponding to the pixel filtering kernel is calculated by normalization using the Softmax function. For each pixel feature in the first image feature, the pixel feature is denoised using the pixel filter kernel and filter kernel weight corresponding to that pixel to obtain the denoised pixel feature of each pixel. Based on the first fusion parameter, the denoised pixel feature of each pixel is fused to obtain the spatial filtering result of the current frame image. If the video to be processed includes a previous frame image adjacent to the current frame image, obtain the second image features obtained by feature extraction of the previous frame image, and obtain the denoised previous frame image obtained by denoising the previous frame image. The second image features, the denoised previous frame image, and the spatial filtering result of the current frame image are fused to obtain a fused image. The fused image is then denoised to obtain the denoised current frame image.

2. The method according to claim 1, characterized in that, The step of extracting features from the current frame image in the video to be processed to obtain a first image feature of the current frame image and a first fusion parameter corresponding to the first image feature includes: Multi-scale feature extraction is performed on the current frame image in the video to be processed to obtain the first image feature and the first fusion parameter at each feature extraction scale; The process involves denoising each pixel feature in the first image feature using the corresponding pixel filter kernel and filter kernel weights to obtain denoised pixel features for each pixel. Then, feature fusion is performed on the denoised pixel features of each pixel based on the first fusion parameters to obtain the spatial filtering result of the current frame image, including: For each pixel feature in the first image feature corresponding to each feature extraction scale, denoising is performed using the pixel filter kernel and filter kernel weight corresponding to the pixel feature to obtain the denoised pixel feature corresponding to each pixel. Based on the first fusion parameter corresponding to each feature extraction scale, feature fusion is performed on the denoised pixel features of each pixel at different feature extraction scales to obtain the spatial filtering result of the current frame image.

3. The method according to claim 2, characterized in that, The feature fusion of pixel features after denoising at different feature extraction scales for each pixel, based on the first fusion parameter corresponding to each feature extraction scale, yields the spatial filtering result of the current frame image, including: The pixel feature ranked second in the feature extraction scale is downsampled to obtain the downsampling result. The downsampling result is then upsampled to obtain the first upsampled feature. The pixel feature ranked first is then upsampled to extract the second upsampled feature. Based on the second-ranked pixel features, the first upsampled features, the second upsampled features, and the first fusion parameters corresponding to the second-ranked pixel features, feature fusion filtering is performed on the second-ranked pixel features to obtain the second-ranked pixel features after spatial filtering. The pixel feature ranked first in the sorting order is deleted, and the pixel feature ranked second in the sorting order of feature extraction scale is downsampled again to obtain the downsampling result. The downsampling result is then upsampled to obtain the first upsampled feature. The pixel feature ranked first is then upsampled to obtain the second upsampled feature. This process continues until the pixel feature ranked second is the last pixel feature in the sorting order. Finally, the pixel feature ranked second is spatially filtered and used as the spatial filtering result of the current frame image.

4. The method according to claim 3, characterized in that, The step of performing feature fusion filtering on the second-ranked pixel features based on the second-ranked pixel features, the first upsampled feature, the second upsampled feature, and the first fusion parameter corresponding to the second-ranked pixel features to obtain spatially filtered pixel features, includes: Multiplying the first upsampled feature with the fusion parameter corresponding to the second-ranked pixel feature yields the filtered noise of the second-ranked pixel feature; multiplying the second upsampled feature with the fusion parameter corresponding to the second-ranked pixel feature yields the canceled noise of the second-ranked pixel feature. Subtract the filtered noise and add the canceled noise from the pixel features ranked second to obtain the spatially filtered pixel features.

5. The method according to claim 2, characterized in that, The process involves multi-scale feature extraction of the current frame image in the video to be processed, obtaining the first image features and the first fusion parameters at each feature extraction scale, including: The trained feature extraction model is used to perform multi-scale feature extraction on the current frame image in the video to be processed, so as to obtain the first image feature of the current frame image at each feature extraction scale and the first fusion parameter corresponding to the first image feature. The trained feature extraction model is trained based on multiple sets of video sample images. Each set of video sample images includes multiple noise sample images belonging to the same video and a denoised sample image corresponding to each noise sample image.

6. The method according to claim 5, characterized in that, Before performing multi-scale feature extraction on the current frame image of the video to be processed using the trained feature extraction model, the method further includes: Multiple sets of video sample images are input into the network model to extract features from the noise samples in each set of video sample images, thereby obtaining the noise sample features and corresponding sample fusion parameters for each noise sample. Based on the sample pixel features included in each pixel of the noise sample features, obtain the sample pixel filter kernel and sample filter kernel weight corresponding to each sample pixel; For each sample pixel in any target noise sample image, the sample pixel features are denoised using the sample pixel filtering kernel and sample filtering kernel weight corresponding to that sample pixel to obtain the denoised sample pixel features corresponding to each sample pixel. Based on the sample fusion parameters, the denoised sample pixel features of each sample pixel are fused to obtain the spatial filtering result of the target noise sample image. If the video sample image to which the target noise sample image belongs includes a previous frame noise sample image adjacent to the target noise sample image, obtain the denoised previous frame noise sample image obtained by denoising the previous frame noise sample image. The sample image features of the previous frame noise sample image, the denoised previous frame sample image, and the spatial filtering result of the target noise sample image are fused to obtain a sample fusion image. The sample fusion image is then filtered to obtain the denoised result of the target noise sample image. Based on the denoising result of the target noise sample image, the denoising result of the previous frame noise sample image, the denoised sample image corresponding to the target noise image, and the denoised sample image corresponding to the previous frame noise sample image, the loss is calculated to obtain the model loss. The model parameters of the network model are adjusted based on the model loss until the model converges, thus obtaining the trained network model.

7. The method according to claim 6, characterized in that, The loss is calculated based on the denoising result of the target noise sample image, the denoising result of the previous frame noise sample image, the denoised sample image corresponding to the target noise image, and the denoised sample image corresponding to the previous frame noise sample image to obtain the model loss, including: Spatial loss is calculated based on the denoising result of the target noise sample image and the denoised sample image corresponding to the target noise image; The time denoising loss is obtained by calculating the pixel feature difference between the denoising result of the target noise sample image and the denoising result of the previous frame noise sample image, and the pixel feature difference between the denoised sample image corresponding to the target noise image and the denoised sample image corresponding to the previous frame noise sample image. The model loss is obtained by weighted summation of the temporal denoising loss and the spatial denoising loss.

8. The method according to claim 1, characterized in that, The step of fusing the second image features, the denoised previous frame image, and the spatial filtering result of the current frame image to obtain the fused image includes: The denoised previous frame image is projected onto the current frame image to obtain the projected previous frame image. The second image feature is then projected onto the current frame image to obtain the projected second image feature. The features of the previous frame image after projection and the second image after projection are concatenated to obtain a concatenated feature map, and the concatenated features are convolved to obtain a temporal feature map. The temporal feature map and the spatial filtering result of the current frame image are linearly fused to obtain a fused image.

9. The method according to claim 8, characterized in that, The step of linearly obtaining a fused image by combining the temporal feature map and the spatial filtering result of the current frame image includes: The fused image is obtained by weighted summation of the temporal feature map and the spatial filtering result of the current frame image, wherein the weight of the temporal feature map is between 0.1 and 0.

2.

10. The method according to claim 1, characterized in that, The step of denoising the fused image to obtain the denoised current frame image includes: Feature extraction is performed on the fused image to obtain fused image features and a second fusion parameter; Based on the feature vector included in each fused image pixel in the fused image features, the fused pixel filter kernel and fused filter kernel weight corresponding to each fused image pixel are obtained; For each fused pixel feature in the fused image features, denoising is performed using the fused pixel filter kernel and the weight of the fused filter kernel corresponding to the fused pixel feature, so as to obtain the denoised fused pixel feature corresponding to each pixel. Based on the second fusion parameter, feature fusion is performed on the fused pixel features after denoising each pixel to obtain the denoised current frame image.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: If the video to be processed does not include a previous frame image adjacent to the current frame image, the spatial filtering result of the current frame image is used as the denoised current frame image.

12. A video image processing apparatus, characterized in that, The device includes: The first data acquisition module is used to extract features from the current frame image in the video to be processed, obtain the first image feature of the current frame image and the first fusion parameter corresponding to the first image feature, map the pixel features of each pixel in the first image feature to two feature maps with the same number of channels through convolution, take the product of the vectors corresponding to the pixel in the two feature maps as the pixel filtering kernel corresponding to the pixel, and use the Softmax function to normalize and calculate the filtering kernel weight corresponding to the pixel filtering kernel; The first filtering module is used to denoise the pixel features of each pixel in the first image features using the pixel filtering kernel and the filtering kernel weight corresponding to the pixel, to obtain the denoised pixel features of each pixel, and to perform feature fusion on the denoised pixel features of each pixel based on the first fusion parameters to obtain the spatial filtering result of the current frame image. The second data acquisition module is used to acquire, when the video to be processed includes a previous frame image adjacent to the current frame image, a second image feature obtained by feature extraction of the previous frame image, and a denoised previous frame image obtained by denoising the previous frame image. The second filtering module is used to fuse the second image features, the denoised previous frame image, and the spatial filtering result of the current frame image to obtain a fused image, and to denoise the fused image to obtain a denoised current frame image.

13. An electronic device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-11.

Images