Image processing methods, model training methods, electronic devices and storage media

Abstract

This application provides an image processing method, a model training method, an electronic device, and a storage medium. The image processing method includes: acquiring color information corresponding to each image frame in a target image frame sequence rendered via Monte Carlo path tracing; determining target color information corresponding to image frame i based on first color information of image frame (i-1) and second color information of image frame i; determining multiple color features and multiple weight parameters corresponding to image frame i based on the target color information; generating multiple grid sampling features based on the first color information and the multiple color features, the multiple grid sampling features corresponding to different image resolutions; decoding the multiple grid sampling features using the multiple color features to obtain multiple grid decoded features, the multiple grid decoded features corresponding to the same image resolution; and determining the denoised color information corresponding to image frame i based on the multiple grid decoded features and the multiple weight parameters.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, a model training method, an electronic device, and a storage medium. Background Technology

[0002] Currently, Monte Carlo path tracing is widely used in film special effects, animation, and games to render images of models in 3D scenes, achieving realistic rendering effects. However, electronic devices often require a certain amount of computing power to render images using Monte Carlo path tracing. Therefore, Monte Carlo path tracing is mostly used on PCs, limiting its application scenarios.

[0003] As the hardware capabilities of mobile devices improve, they can now provide the necessary computational power for Monte Carlo path tracing. However, due to the limited computing power available to these devices, they often cannot maintain rendering speed at the expense of rendering quality, resulting in images with significant noise. Therefore, to render realistic images using Monte Carlo path tracing on mobile devices, it is essential to provide an image denoising method that matches the computing power of these devices. Summary of the Invention

[0004] This application provides an image processing method, a model training method, an electronic device, and a storage medium for denoising images rendered using Monte Carlo path tracing on a mobile electronic device.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] Obtain the target image frame sequence rendered by Monte Carlo path tracing, and the color information corresponding to each image frame in the target image frame sequence;

[0007] The target color information corresponding to the target image frame is determined based on the first color information corresponding to the previous image frame and the second color information corresponding to the target image frame. The target image frame is any image frame in the target image frame sequence except for the first image frame.

[0008] Based on the target color information, determine multiple color features corresponding to the target image frame, and weight parameters corresponding to the multiple color features when representing the target color information;

[0009] Based on the first color information and the plurality of color features, a plurality of grid sampling features are generated that correspond one-to-one with the plurality of color features, and the plurality of grid sampling features correspond to different image resolutions respectively;

[0010] Decode the multiple grid sampling features based on the multiple color features to obtain multiple grid decoding features corresponding to the multiple grid sampling features, wherein the multiple grid decoding features correspond to the same image resolution;

[0011] Based on the weight parameters corresponding to the multiple grid decoding features and the multiple color features, the denoised color information corresponding to the target image frame is determined.

[0012] Secondly, embodiments of this application also provide an image denoising model training method, including:

[0013] Obtain a training sample set, which includes: a first image frame sequence and a second image frame sequence rendered by Monte Carlo path tracing for the same object, wherein the image noise in the first image frame sequence is greater than that in the second image frame sequence;

[0014] The color information corresponding to each image frame in the first image frame sequence and the color information corresponding to each image frame in the second image frame sequence are input into the image denoising model, so that the image denoising model performs the following training operation:

[0015] Based on the first color information corresponding to the previous image frame and the second color information corresponding to the first target image frame, the target color information corresponding to the first target image frame is determined. The first target image frame is any image frame in the first image frame sequence except for the first image frame.

[0016] Based on the target color information, determine multiple color features corresponding to the first target image frame, and weight parameters corresponding to the multiple color features respectively when representing the target color information;

[0017] Based on the first color information and the plurality of color features, a plurality of grid sampling features are generated that correspond one-to-one with the plurality of color features, and the plurality of grid sampling features correspond to different image resolutions respectively;

[0018] Decode the multiple grid sampling features based on the multiple color features to obtain multiple grid decoding features corresponding to the multiple grid sampling features, wherein the multiple grid decoding features correspond to the same image resolution;

[0019] Based on the weight parameters corresponding to the multiple grid decoding features and the multiple color features, the denoised color information corresponding to the first target image frame is determined;

[0020] Determine the second target image frame in the second image frame sequence that corresponds to the first target image frame;

[0021] Calculate the loss value between the denoised color information corresponding to the first target image frame and the color information corresponding to the second target image frame;

[0022] If the loss value is greater than a set threshold, the model parameters of the image denoising model are adjusted according to the loss value.

[0023] Thirdly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, it implements the image processing method described in the first aspect or the image denoising model training method described in the second aspect.

[0024] Fourthly, embodiments of this application also provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image processing method described in the first aspect or the image denoising model training method described in the second aspect.

[0025] This application provides a lightweight image denoising model for mobile electronic devices. This model denoises a sequence of target image frames rendered using Monte Carlo path tracing as follows: First, it acquires the color information corresponding to each image frame in the target image frame sequence. Then, for any image frame (i.e., the target image frame) excluding the first image frame, it performs frame accumulation on the color information of different frames based on the first color information corresponding to the preceding image frame and the second color information corresponding to the target image frame, to determine richer target color information corresponding to the target image frame after frame accumulation. Next, it extracts features from the target color information to determine multiple color features corresponding to the target image frame, and weight parameters corresponding to each color feature when representing the target color information. Finally, it maps the first color information and multiple color features onto a grid, generating multiple grid sampling features that correspond one-to-one with the multiple color features. Each grid sampling feature is obtained based on different grid sampling parameters, and the resulting multiple grid sampling features correspond to different image resolutions. After obtaining multiple grid sampling features corresponding to different image resolutions, these features are decoded based on multiple color features to obtain multiple grid decoded features. These multiple grid decoded features share the same feature scale, meaning they correspond to the same image resolution. Finally, based on the weight parameters corresponding to the multiple grid decoded features and multiple color features, the denoised color information of the target image frame is determined.

[0026] In this scheme, multiple grid sampling features with different feature scales (i.e. image resolutions) are generated to achieve image denoising of the target image frame at different image granularities. Since the image information corresponding to each grid in the grid sampling features corresponding to different image resolutions is different, when multiple grid decoding features are fused after decoding multiple grid sampling features, richer image semantic information can be obtained, and the denoised color information corresponding to the target image frame can be determined more accurately. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 This is a schematic diagram of the structure of a mobile electronic device provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of a software module architecture provided in an embodiment of this application;

[0030] Figure 3 This application provides an illustration of an image processing method provided in an embodiment of the present application. Figure 1 ;

[0031] Figure 4 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0032] Figure 5 This application provides an illustration of an image processing method provided in an embodiment of the present application. Figure 2 ;

[0033] Figure 6 This is a schematic diagram of a scenario for determining target color information provided in an embodiment of this application;

[0034] Figure 7 A schematic diagram of a feature extraction scenario provided in an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of a grid sampling scenario provided in an embodiment of this application;

[0036] Figure 9 This is a schematic diagram of a grid decoding scenario provided in an embodiment of this application;

[0037] Figure 10 This is a schematic diagram of the model structure of an image denoising model provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0040] Currently, in fields such as film special effects, animation, and games that require image rendering of 3D scenes, commonly used image rendering methods include rasterization rendering and path tracing. Among them, path tracing can simulate the propagation of real light in a 3D scene and calculate lighting effects more accurately, thus its rendering effect is more realistic than rasterization rendering.

[0041] Monte Carlo path tracing rendering is a type of path tracing. Its basic idea is as follows: starting from the viewpoint (i.e., the virtual camera), a virtual ray is emitted into the 3D scene through each pixel on the projection screen; when the ray intersects the surface of an object in the 3D scene, it is selectively and randomly reflected in other directions according to the material properties of the object surface. This process is iterated until the ray hits the light source or escapes from the 3D scene; finally, Monte Carlo integration is used to determine the color value corresponding to the pixel based on the contribution of each ray to the pixel.

[0042] In practical applications, the process of controlling the number of rays emitted by each pixel to determine the color of each pixel is called sampling. This can be represented by Sample Per Pixel (SPP), which indicates the number of rays emitted by each pixel, i.e., the number of samples. For example, if each pixel emits one ray, the sampling number is considered to be 1, denoted as 1SPP. A larger SPP value results in more accurate color values ​​for the pixel determined using Monte Carlo integration, and less noise in the rendered image. However, a larger SPP value requires more rays for path tracing, increasing the computational burden on the electronic device's hardware.

[0043] With the advancement of hardware capabilities in mobile devices, Monte Carlo path tracing is no longer limited to PCs; mobile devices can now provide the necessary computational power for it. However, due to the limited computing power of mobile devices, they support fewer SPPs (Sessions Per Pixel) compared to PCs for the same rendering time. For example, a PC can easily support 8, 16, or even 24 SPPs of rendering within the same timeframe, while a mobile device can only support 1 SPP. This means that images rendered using Monte Carlo path tracing on mobile devices often contain significant noise. Therefore, denoising is essential for images rendered using Monte Carlo path tracing with low SPP support on mobile devices.

[0044] Generally, denoising methods can be divided into two categories: traditional operator-based methods, such as Spatiotemporal Variance Guided Filtering (SVGF), and deep learning methods based on neural networks, such as Neural Bilateral Grid (NBG). Operator-based methods often cannot handle complex scene variations; therefore, deep learning methods based on neural networks are frequently used for image denoising.

[0045] However, existing neural network-based deep learning methods are often developed for PCs, which presents challenges such as high resource consumption and difficulties in mobile deployment. Furthermore, mobile hardware may not support these methods. For example, in NBG, the GuideNet structure is a five-layer DenseNet structure, and the excessive number of branches can lead to memory overflow on mobile devices. Additionally, the Grid Creation and Grid Slicing operations in NBG are custom CUDA operators, which are not supported by mobile GPUs. Even if these CUDA operators are converted to operations supported by mobile devices, the cumulative effect can still easily lead to memory overflow.

[0046] Therefore, in order to enable mobile electronic devices to render realistic images using Monte Carlo path tracing, it is essential to provide an image denoising method that matches the computing power of mobile electronic devices. To this end, embodiments of this application provide an image processing method for denoising. This method can be implemented by constructing a lightweight neural network model, aiming to denoise images rendered using low-SPP Monte Carlo path tracing supported by mobile electronic devices with minimal resource consumption.

[0047] The image processing method provided in this application can be applied to mobile electronic devices, including but not limited to embedded devices with user interfaces such as mobile phones, tablets, laptops, wearable devices, and televisions.

[0048] Figure 1 This is a schematic diagram of the structure of a mobile electronic device provided in an embodiment of this application. This application uses a mobile phone as an example to describe the mobile electronic device 100 provided in this application.

[0049] like Figure 1 As shown, the mobile electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0050] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile electronic device 100. In other embodiments of this application, the mobile electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0051] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a baseband processor. Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0052] The processor 110 may also include a memory for storing instructions and data.

[0053] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0054] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces.

[0055] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to realize communication between the processor 110 and the audio module 170.

[0056] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface.

[0057] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160.

[0058] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. In some embodiments, the processor 110 and the camera 193 communicate via the MIPI interface to enable the shooting function of the mobile electronic device 100. The processor 110 and the display screen 194 communicate via the MIPI interface to enable the display function of the mobile electronic device 100.

[0059] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc.

[0060] USB interface 130 is an interface that conforms to the USB standard specification. USB interface 130 can be used to connect a charger to charge the mobile electronic device 100, and can also be used for data transfer between the mobile electronic device 100 and peripheral devices.

[0061] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the mobile electronic device 100.

[0062] The charging management module 140 receives charging input from a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 can also supply power to electronic devices via the power management module 141.

[0063] The wireless communication function of the mobile electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0064] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0065] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on mobile electronic devices 100.

[0066] The wireless communication module 160 can provide solutions for wireless communication applications on the mobile electronic device 100, including Wireless Local Area Networks (WLAN), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR). The wireless communication module 160 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signal, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0067] In some embodiments, antenna 1 of mobile electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling mobile electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), etc.

[0068] The mobile electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a graphics processing unit that connects the display screen 194 and the application processor.

[0069] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), or the like. In some embodiments, the mobile electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0070] The mobile electronic device 100 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 194, and application processor. The ISP is used to process the data fed back by the camera 193.

[0071] Camera 193 is used to capture still images or videos. In some embodiments, mobile electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0072] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0073] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of mobile electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0074] Mobile electronic device 100 can implement audio functions such as music playback and recording through an audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0075] The audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal.

[0076] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The mobile electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0077] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the mobile electronic device 100 answers a phone call or voice message, it can listen to the voice by bringing the receiver 170B close to the user's ear.

[0078] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. The mobile electronic device 100 may be equipped with at least one microphone 170C. In some embodiments, the mobile electronic device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, the mobile electronic device 100 may be equipped with three, four, or more microphones 170C, enabling sound signal collection, noise reduction, sound source identification, and directional recording functions, etc.

[0079] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface, a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0080] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194.

[0081] The gyroscope sensor 180B can be used to determine the motion posture of the mobile electronic device 100.

[0082] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the mobile electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0083] The magnetic sensor 180D includes a Hall sensor. The mobile electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the mobile electronic device 100 is a flip phone, the mobile electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D.

[0084] The 180E accelerometer can detect the magnitude of acceleration in various directions (typically three axes) of a mobile electronic device 100. It can also be used to identify the posture of the electronic device, and is applied to applications such as screen orientation switching and pedometers.

[0085] A distance sensor 180F is used to measure distance. The mobile electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the mobile electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0086] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a photodetector. The mobile electronic device 100 uses the photodiode to detect infrared reflected light from a nearby object. When sufficient reflected light is detected, it can be determined that there is an object near the mobile electronic device 100; otherwise, there is no object near the mobile electronic device 100.

[0087] An ambient light sensor 180L is used to sense ambient light intensity. The mobile electronic device 100 can adaptively adjust the brightness of its display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also work in conjunction with a proximity sensor 180G to detect whether the mobile electronic device 100 is in a pocket, preventing accidental touches.

[0088] The fingerprint sensor 180H is used to collect fingerprints. The mobile electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, app access lock, fingerprint photography, fingerprint answering of incoming calls, etc.

[0089] Temperature sensor 180J is used to detect temperature. In some embodiments, mobile electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy. In some embodiments, when the temperature is below another threshold, mobile electronic device 100 heats battery 142 or boosts the output voltage of battery 142 to prevent low temperature from causing abnormal shutdown of mobile electronic device 100.

[0090] Touch sensor 180K, also known as a "touch device," can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K is used to detect touch operations applied to or near it. In other embodiments, touch sensor 180K may also be disposed on the surface of mobile electronic device 100, in a different location than display screen 194.

[0091] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. The mobile electronic device 100 can receive button inputs and generate key signal inputs related to user settings and function control of the mobile electronic device 100.

[0092] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, or to indicate messages, missed calls, notifications, etc.

[0093] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the mobile electronic device 100. The mobile electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0094] The image processing methods described in the following embodiments can all be implemented in a mobile electronic device 100 having the above-described hardware structure.

[0095] The software system of the aforementioned mobile electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the mobile electronic device 100.

[0096] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through interfaces. In some embodiments, the Android system may include an application layer, an application framework layer, an Android runtime and system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application uses the Android system as an example; however, the solution can also be implemented in other operating systems (such as HarmonyOS, iOS, etc.) as long as the functions implemented by each module are similar to those in the embodiments of this application.

[0097] The application layer can include a series of application packages.

[0098] Figure 2 This is a schematic diagram of a software module architecture provided for an embodiment of this application.

[0099] like Figure 2As shown, the application package may include applications such as camera, gallery, calendar, call, map, video application, game application, and VR / AR application. Of course, the application layer may also include other application packages, such as payment application, shopping application, and banking application, etc., which are not limited in this application.

[0100] Video applications, game applications, and VR / AR applications can be native applications that come with the operating system or third-party applications. Video applications can be used to play and edit videos; game applications are used to provide game services; VR / AR applications can be understood as applications that support virtual reality or augmented reality technologies. In some embodiments, when a video application, game application, or VR / AR application needs to display an image during operation, the image can be rendered using the image processing method provided in the embodiments of this application.

[0101] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, it may include a window manager, content provider, resource manager, notification manager, etc., but this embodiment does not impose any limitations on these.

[0102] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0103] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0104] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0105] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), and rendering engines.

[0106] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0107] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0108] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0109] A 2D graphics engine is a graphics engine for 2D drawing.

[0110] Rendering refers to the process of generating images from a model using software. The model is a description of a three-dimensional object (or object, 3D model, or model) using a language or data structure, and includes geometry, viewpoint, texture, and lighting information. Images can include digital images or bitmap images. A rendering engine can be understood as software that uses rendering algorithms to generate images from a model. In some embodiments, the rendering engine may be located on a graphics processing unit (GPU).

[0111] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that Java calls, and the other part contains the core Android libraries.

[0112] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0113] The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, sensor drivers, etc. In some embodiments, the display driver is used to control the display screen to show images; the sensor driver is used to control the operation of multiple sensors, such as controlling the operation of pressure sensors and touch sensors.

[0114] The hardware layer may include the aforementioned electronic device hardware components. For example, Figure 2 The display and GPU were showcased. The GPU may include a path tracing rendering pipeline.

[0115] The following uses a game application at the application layer as an example to introduce one application scenario of the image processing method provided in this application embodiment.

[0116] Figure 3 This application provides an illustration of an image processing method provided in an embodiment of the present application. Figure 1 .

[0117] like Figure 3 As shown, when a game application needs to display an image, it can send the model data of the 3D scene to the rendering engine through the rendering framework of the application framework layer. The model data of the 3D scene may include: the geometric information and material information of each model in the 3D scene, as well as the path tracing rendering pipeline identifier corresponding to each model.

[0118] The rendering engine receives model data of a 3D scene, renders the 3D scene image using Monte Carlo path tracing, and then uses the image processing method provided in this application embodiment to denoise the 3D scene image rendered by Monte Carlo path tracing, obtaining a denoised 3D scene image. The rendering engine can then upload the denoised 3D scene image to the game application through the rendering framework of the application framework layer.

[0119] When a game application receives a rendered image, if it needs to display the denoised 3D scene image, it sends the image data of the denoised 3D scene image to the display driver through the application framework layer. The display driver then controls the display screen to show the denoised 3D scene image.

[0120] The image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Then, the training process of the image denoising model used to implement the image processing method will be explained.

[0121] Figure 4 A flowchart of an image processing method provided in an embodiment of this application is shown below. Figure 4 As shown, the image processing method includes the following steps:

[0122] 401. Obtain the target image frame sequence obtained by Monte Carlo path tracing rendering, and the color information corresponding to each image frame in the target image frame sequence.

[0123] 402. Based on the first color information corresponding to the previous image frame and the second color information corresponding to the target image frame, determine the target color information corresponding to the target image frame. The target image frame is any image frame in the target image frame sequence except for the first image frame.

[0124] 403. Based on the target color information, determine multiple color features corresponding to the target image frame, and the weight parameters corresponding to the multiple color features when representing the target color information.

[0125] 404. Based on the first color information and multiple color features, generate multiple grid sampling features that correspond one-to-one with the multiple color features, and the multiple grid sampling features correspond to different image resolutions.

[0126] 405. Decode multiple grid sampling features based on multiple color features to obtain multiple grid decoding features corresponding to the multiple grid sampling features, and the multiple grid decoding features correspond to the same image resolution.

[0127] 406. Determine the denoised color information of the target image frame based on the weight parameters corresponding to multiple grid decoding features and multiple color features.

[0128] The image processing method provided in this embodiment is applied to a scenario in which multiple consecutive frames of images are rendered using low SPP Monte Carlo path tracing. The object of image processing is the target image frame sequence obtained by Monte Carlo path tracing rendering. The goal of image processing is to determine the correct color information corresponding to each image frame in the target image frame sequence by using a small amount of resources, that is, to perform noise reduction processing on the target image frame sequence.

[0129] The target image frame sequence includes several image frames, such as image frame 1, image frame 2, ..., image frame N (N is a positive integer), where the N image frames are arranged in chronological order. After the target image frame sequence is obtained through Monte Carlo path tracing rendering, each image frame in the target image frame sequence carries its corresponding color information, which is used for subsequent image denoising processing.

[0130] Optionally, the color information corresponding to each image frame includes: the albedo, pixel depth, pixel normal, and color after removing the albedo for each pixel in each image frame. Here, albedo is a physical quantity representing the ability of an object to reflect light; pixel depth is the depth of the pixel position relative to the virtual camera during Monte Carlo path tracing rendering; pixel depth and pixel normal serve to calibrate the pixel position.

[0131] In image processing based on the color information corresponding to each image frame, since the processing procedure is the same for any image frame, taking the target image frame (i.e., image frame i, where i can be 2, 3, ..., or N) in the target frame sequence as an example, combined with... Figure 5 The image processing method provided in this embodiment will be described. Figure 5 This application provides an illustration of an image processing method provided in an embodiment of the present application. Figure 2 .

[0132] For ease of description, the target image frame is referred to as image frame i, the previous image frame is referred to as image frame (i-1), the color information of image frame (i-1) is referred to as the first color information, and the color information of image frame i is referred to as the second color information.

[0133] To improve the image denoising effect and ensure the accuracy of the color information after denoising image frame i, firstly, the color information of image frame i and image frame (i-1) is processed by frame accumulation, that is, the target color information corresponding to image frame i is determined based on the first color information and the second color information.

[0134] Figure 6 This is a schematic diagram of a scene for determining target color information provided in an embodiment of this application, such as... Figure 6 As shown, as an optional frame accumulation implementation, the first position of the virtual camera when rendering image frame (i-1) via Monte Carlo path tracing, and the second position of the virtual camera when rendering image frame i via Monte Carlo path tracing, can be obtained first. Both the first and second positions can be described by the position matrix corresponding to the virtual camera. Then, based on the first and second positions, the first color information is mapped to the pixel coordinate system corresponding to the second color information to obtain the third color information corresponding to the first color information. For example, using the position matrix corresponding to the first and second positions of the virtual camera, a position transformation matrix from the first position to the second position is determined. Based on this position transformation matrix, a reprojection algorithm is used to map the first color information to the pixel coordinate system corresponding to the second color information, and the mapping information of the first color information in the pixel coordinate system corresponding to the second color information is recorded as the third color information. Finally, based on the third color information and the second color information, the target color information corresponding to the target image frame is determined.

[0135] Optionally, determining the target color information corresponding to image frame i based on the third color information and the second color information includes: calculating a weighted average of the colors containing albedo removed in the third color information and the colors containing albedo removed in the second color information based on preset weighting coefficients; calculating a first dot product of the weighted average and the illuminance contained in the second color information; and using the first dot product and the albedo, pixel depth, and pixel normal contained in the second color information as the target color information corresponding to image frame i.

[0136] The preset weighting coefficients describe the respective weights of the colors in the third color information (after removing albedo) and the second color information (after removing albedo) during frame accumulation calculations. For example, the weighting coefficient for the colors in the third color information (after removing albedo) is 0.2, and the weighting coefficient for the colors in the second color information (after removing albedo) is 0.8. Optionally, these weighting coefficients can be determined based on the denoising effects of multiple image processing steps.

[0137] After determining the target color information corresponding to image frame i, the next step is to extract features from the target color information so that subsequent image denoising operations can be performed based on the extracted features.

[0138] Figure 7 This is a schematic diagram of a feature extraction scenario provided in an embodiment of this application, such as... Figure 7 As shown, as an optional feature extraction method, the target color information can first be regularized and channel concatenated to determine the first color feature corresponding to the target color information; then, the number of channels of the first color feature can be adjusted to the target number of channels through operations such as convolution to obtain the second color feature; finally, the second color feature is sliced ​​according to channels to determine multiple color features corresponding to image frame i. Figure 7 The diagram illustrates color feature 1, color feature 2, ..., color feature m, where m is a positive integer, and the weight parameters corresponding to each color feature when representing target color information. Figure 7 The diagram illustrates weight parameter 1, weight parameter 2, ..., weight parameter m.

[0139] The number of multiple color features can be flexibly configured according to the computing power of the mobile electronic device. The number of multiple color features and the number of weight parameters are matched with the number of target channels. For example, the sum of the number of multiple color features and the number of weight parameters is equal to the number of target channels, or the sum of the number of multiple color features and the number of weight parameters minus 1 is equal to the number of target channels.

[0140] Optionally, during feature extraction, the number of convolution operations, the kernel size, the stride, and the padding can all be flexibly configured according to the computing power of the mobile electronic device.

[0141] To ensure that multiple color features and their corresponding weights remain consistently positive, and to guarantee that the final color calculation is meaningful, the output value is constrained to between 0 and 1 after each convolution operation using an activation function. These activation functions include, but are not limited to, ReLU, Leaky ReLU, PreLU, Sigmoid, tanh, and SELU.

[0142] After obtaining multiple color features and their corresponding weight parameters through feature extraction, the system further generates multiple grid sampling features at different feature scales (i.e., image resolutions) based on the multiple color features and the third color feature, thereby achieving image denoising of image frame i at different image granularities.

[0143] Figure 8 This is a schematic diagram of a grid sampling scenario provided in an embodiment of this application, such as... Figure 8 As shown, as an optional grid sampling method, the color containing albedo removed from the third color information can be first concatenated with multiple color features (color feature 1, color feature 2, ..., color feature m) to obtain multiple channel concatenation results. Figure 8 The diagram illustrates channel connection results 1, 2, ..., m. Then, multiple channel connection results are subjected to grid sampling through operations such as convolution and max pooling to obtain multiple grid sampling features. Figure 8 The grid sampling feature shown in the diagram is 1, 2, ..., m).

[0144] In this process, the grid sampling parameters differ for each channel connection result. Optionally, the grid sampling parameters include: downsampling coefficients and channel adjustment coefficients.

[0145] In the aforementioned grid sampling process, different color features are sampled using different downsampling coefficients and channel adjustment coefficients. This allows for the filtering of image noise at different granularities, resulting in grid sampling features corresponding to different feature sizes (i.e., different resolutions). Since each feature corresponds to a different image region (i.e., image patch), fusing multiple grid sampling features can yield richer image semantic information and improve denoising performance.

[0146] For multiple grid sampling features with different feature sizes, it is necessary to decode the multiple grid sampling features in order to facilitate feature fusion.

[0147] Figure 9 This is a schematic diagram of a grid decoding scenario provided in an embodiment of this application, such as... Figure 9 As shown, as an optional grid decoding method, firstly, multiple grid sampling features (grid sampling feature 1, grid sampling feature 2, ..., grid sampling feature m) are upsampled using methods such as deconvolution. The upsampled grid sampling features correspond to the same image resolution, i.e., have the same feature size. It is understood that different grid sampling features correspond to different upsampling coefficients. Next, the second dot product of the upsampled grid sampling features with their corresponding color features is calculated. That is, the multiple grid sampling features are decoded using multiple color features to determine the multiple grid decoding features corresponding to the upsampled grid sampling features. Figure 9 The diagram illustrates grid decoding feature 1, grid decoding feature 2, ..., grid decoding feature m. Multiple grid decoding features correspond to the same image resolution.

[0148] Optionally, to ensure that the decoded features of multiple grids are consistently positive and that the final color calculation is meaningful, the output value is restricted to between 0 and 1 after the deconvolution operation using an activation function. Activation functions include, but are not limited to, ReLU, LeakyReLU, PreLU, Sigmoid, tanh, and SELU.

[0149] Finally, multiple grid decoding features are fused. In the specific implementation, based on the first correspondence between the multiple grid decoding features and multiple color features, and the second correspondence between the multiple color features and weight parameters, the weight parameters corresponding to each of the multiple grid decoding features are determined. Based on the multiple grid decoding features and their respective weight parameters, a weighted average of the multiple grid decoding features is determined. Based on the weighted average, the denoised color information corresponding to image frame i is determined.

[0150] To make it easier to understand, let's take an example. Figures 5 to 9 Taking the illustrated scenario as an example, if grid decoding feature 1 corresponds to color feature 1, and color feature 1 corresponds to weight parameter 1, then grid decoding feature 1 corresponds to weight parameter 1. Similarly, grid decoding feature 2 corresponds to weight parameter 2, ..., grid decoding feature m corresponds to weight parameter m. Then, the weighted average of the m grid decoding features is determined as (grid decoding feature 1 ☉ weight parameter 1) + (grid decoding feature 2 ☉ weight parameter 2) + ... + (grid decoding feature m ☉ weight parameter m). Finally, the denoised color corresponding to image frame i is determined as the product of the weighted average of the m grid decoding features and the albedo in the target color information.

[0151] The above is a detailed description of the image processing method provided in the embodiments of this application. For ease of understanding, the image processing method provided in the embodiments of this application will be illustrated below by substituting specific parameter values. It should be noted that the specific parameter values ​​are for illustrative purposes only and are not intended to be limiting.

[0152] Assume the acquired target image frame sequence S contains N image frames, each image frame has a low SPP (e.g., 1SPP-8SPP), the feature size of each image frame is h×w, and the color information of each frame image includes: the albedo a∈R of each pixel. h×w×3 Pixel depth d∈R h×w×1 Pixel normal p∈R h×w×3 and the color c∈R after removing albedo h×w×3 Then the color information of each image frame can be represented by the feature r = (a, d, p, c), where r ∈ R. h ×w×10 Correspondingly, the target image frame sequence S can be represented as S={(r1,t1),…,(r N ,t N )}, where t is used to represent its timing.

[0153] 1) Determine the target color information corresponding to image frame i

[0154] When determining the target color information corresponding to image frame i through frame accumulation, the color information r of image frame (i-1) is first... i-1 Color information r mapped to image frame i i The corresponding pixel coordinate system is used to obtain r. i-1 The mapping result r' i-1 Among them, r i =(a i ,d i ,p i ,c i ), r i-1 =(a i-1 ,d i-1 ,p i-1 ,c i-1 ), r' i-1 =(a' i-1 ,d' i-1 ,p' i-1 ,c' i-1 ).

[0155] Then, based on the preset weighting coefficients, r' is calculated. i-1 The color c' included for removing albedo i-1 With r i The included color c for removing albedo i The weighted average result Ci = Avg(c' i-1 ,c i ).

[0156] Next, the color Ci and albedo a will be removed. i The color information n = a is obtained by performing element-wise matrix multiplication (i.e., dot product). i ⊙Ci. Where n can also be regarded as the imaging result of image frame i after frame accumulation and before removing image noise.

[0157] Finally, feature r' i =(a i ,d i ,p i ,n) serves as the target color information corresponding to image frame i.

[0158] 2) Extract features from target color information

[0159] First, (a) i ,d i ,p i Regularization (,n) is performed, and channel concatenation is applied to determine the first color feature f, where f = cat(normalize(a,n)). i ,d i ,p i ,n)),f∈Rh×w×10 .

[0160] In this embodiment, assuming the number of color features set during image denoising model training is 3, the number of target channels matching the number of color features is 5. Of these 5 channels, 3 channels correspond to the 3 color features, and the remaining 2 channels correspond to the weight parameters of the first and second color features. It can be understood that the weight parameter of the third color feature can be obtained by subtracting the weight parameters of the first and second color features from 1.

[0161] Suppose we perform a convolution operation using two 5×5 convolutions, adjusting the first color feature f∈R. h×w×10 The number of channels is 10, and the target number of channels is 5, in order to obtain the second color feature f”∈R h×w×5 In this case, the stride of the two 5×5 convolutions is 1, the padding is 2, the output channels of the first convolution are k1=20, and the output channels of the second convolution are k2=5. The activation function after the first convolution is Leaky ReLU, and the activation function after the second convolution is Sigmoid. Therefore, the shape change of the first color feature f during the two convolutions is as follows:

[0162] f∈R h×w×10 →f'∈R h×w×20 →f”∈R h×w×5

[0163] Then, according to the channel pair, the second color feature f”∈R h×w×5 Perform feature slicing processing, that is:

[0164] slice(f”)=(p1,p2,p3,w1,w2)

[0165] Where (p1,p2,p3,w1,w2)∈R h×w×1 p1, p2, and p3 are the three color features corresponding to image frame i, and w1 represents the target color information (a i ,d i ,p i When p1 corresponds to n, w2 is the weight parameter representing the target color information (a i ,d i ,p i The weight parameter corresponding to p2 when n) represents the target color information (a i ,d i ,p i When n), the weight parameter corresponding to p3 is (1-w1-w2).

[0166] 3) Grid sampling

[0167] Assuming that for the above three color features, the three grid sampling parameters are set as follows: p1 corresponds to downsampling coefficient ss = 4 and channel adjustment coefficient sr = 4; p2 corresponds to downsampling coefficient ss = 8 and channel adjustment coefficient sr = 8; p3 corresponds to downsampling coefficient ss = 16 and channel adjustment coefficient sr = 16.

[0168] When performing grid sampling, the processing procedures for p1, p2 and p3 are the same. Taking p1 as an example, we will explain the generation process of multiple grid sampling features corresponding to multiple color features.

[0169] First, the third color information r' i-1 =(a' i-1 ,d' i-1 ,p' i-1 ,c' i-1 The color c' to remove albedo in ) i-1 Connect to p1 via a channel.

[0170] Next, the channel concatenation result is subjected to a coordinate convolution of, for example, 3×3, resulting in an output channel count of 256 / sr, or 256 / 4, yielding the feature f”’=conv(cat(c’ i-1 p1,x,y), where x and y represent the matrices formed by the x and y coordinates in the coordinate convolution, respectively.

[0171] Finally, the grid sampling feature g1 corresponding to p1 is obtained by max pooling the feature f”'.

[0172] in, That is to say

[0173] By analogy, we can obtain the grid sampling feature g2 corresponding to p2. And the grid sampling feature g3 corresponding to p3, As can be seen, the three mesh features g1, g2 and g3 mentioned above correspond to different feature sizes and channels.

[0174] 4) Mesh Decoding

[0175] In the grid decoding stage, p1 decodes g1, p2 decodes g2, and p3 decodes g3. Since the decoding process for any grid sampling feature is the same, we will take decoding g1 with p1 as an example to illustrate the process of decoding multiple grid sampling features to obtain multiple grid decoded features.

[0176] First, g1 is subjected to a coordinate deconvolution with a kernel size of ss×ss (i.e., 4×4), a stride of ss=4, and an output channel of 1. This is then passed through a sigmoid activation function to output the feature f””. The process of determining f”” can be expressed as follows:

[0177]

[0178] Where, f””∈R h×w×1 x and y represent the matrices formed by the x and y coordinates in the coordinate convolution, respectively.

[0179] Then, perform element-wise matrix multiplication (i.e., dot product) between p1 and f”” to obtain the grid decoding feature df1 = f””⊙p1, where df1∈R h×w×1 .

[0180] By analogy, we can obtain the decoding feature df2 corresponding to p2 decoding g2, where df2∈R h×w×1 ; p3 decodes the decoding feature df3 corresponding to g3, where df3∈R h×w×1 .

[0181] 5) Fusion mesh decoding features

[0182] First, based on the correspondence between df1, df2, and df3 and p1, p2, and p3, and the relationship between p1, p2, and p3 and w1, w2, and (1-w1-w2), we determine that the weight parameter corresponding to df1 is w1, the weight parameter corresponding to df2 is w2, and the weight parameter corresponding to df3 is (1-w1-w2).

[0183] Then, a weighted average is calculated for df1, df2, and df3 to generate the denoised color information cn after removing albedo. The calculation process can be expressed as follows:

[0184] cn=(df1⊙w1)+(df2⊙w2)+(df3⊙(1-w1-w2))

[0185] Where ⊙ denotes the dot product, and cn∈R h×w×1 .

[0186] Finally, cn is compared with the color information r' of image frame i. i =(a i ,d i ,p i albedo a in n) i Multiplying these together, we get the denoised color containing the albedo: res = cn × a i This completes the denoising of image frame i.

[0187] In summary, the image processing method provided in this embodiment generates multiple grid sampling features with different feature scales (i.e., image resolutions) to achieve image denoising of the target image frame at different image granularities. Since the image information corresponding to each grid in the grid sampling features for different image resolutions is different, when fusing multiple grid decoding features after decoding multiple grid sampling features, richer image semantic information can be obtained, and the denoised color information corresponding to the target image frame can be determined more accurately. Furthermore, since the image processing method provided in this embodiment involves relatively little computation, even mobile electronic devices can provide sufficient computing power to execute this image processing method.

[0188] It is worth noting that the image processing method described in the foregoing embodiments uses multiple image frames as an example. In fact, the image processing method provided in this embodiment can also be used for denoising a single-frame image; it simply requires replacing the color corresponding to the albedo removed after frame accumulation of image frame i and image frame (i-1) with the color corresponding to the albedo removed from image frame i. When performing single-frame image denoising, adjacent image frames are no longer correlated.

[0189] The image processing method provided in this application can also be deployed on electronic devices with high computing power, such as PCs. In practical applications, since the computing power of PCs is stronger than that of mobile electronic devices, the execution time of this image processing method will be shortened, thereby enabling the execution of image denoising tasks with higher real-time requirements.

[0190] The image processing method provided in this application embodiment can be used not only for denoising images rendered by Monte Carlo path tracing, but also for denoising images rendered by non-Monte Carlo path tracing. It only requires modifying the number of channels in the grid sampling and grid decoding process (for example, changing 10 channels to 3 channels). The modified image processing method can be used for image defogging, deraining and other denoising scenarios.

[0191] The image processing method described above can be implemented using a small-scale image denoising network model. The training process of the image denoising network model is explained below.

[0192] First, a training sample set is obtained, which includes a first image frame sequence and a second image frame sequence rendered by Monte Carlo path tracing for the same object, wherein the image noise in the first image frame sequence is greater than that in the second image frame sequence.

[0193] Specifically, the first and second image frame sequences can be two image frame sequences obtained by Monte Carlo path tracing rendering with different SPPs for the same target scene. The SPP corresponding to the second image frame sequence is much larger than that of the first image frame sequence; for example, the SPP of the second image frame sequence is 4096, while the SPP of the first image frame sequence is 1. The second image frame sequence can be considered as the better result after image denoising, i.e., the label during the training process of the image denoising model.

[0194] Then, the color information corresponding to each image frame in the first image frame sequence and the color information corresponding to each image frame in the second image frame sequence are input into the image denoising model so that the image denoising model can perform training operations.

[0195] To facilitate understanding, the network structure of an image denoising network will be used as an example for explanation. Figure 10 This is a schematic diagram of the model structure of an image denoising model provided in an embodiment of this application. For example... Figure 10 As shown, the image denoising model includes: a data preprocessing module, a backbone network, a mesh generation network, a mesh decoding network, and a fusion module.

[0196] The mesh generation network consists of multiple mesh generation networks with different granularities (i.e., corresponding to different mesh sampling parameters) (e.g., ...). Figure 10 The diagram illustrates mesh generation networks 1, 2, ..., m; each mesh generation network corresponds to a mesh decoding network, therefore, the mesh decoding network also consists of multiple different mesh decoding networks (e.g., ...). Figure 10 The diagram illustrates mesh decoding network 1, mesh decoding network 2, ..., mesh decoding network m.

[0197] During the training of the image denoising network, the data preprocessing module is used to determine the target color information corresponding to the first target image frame based on the first color information corresponding to the preceding image frame and the second color information corresponding to the first target image frame. The first target image frame is any image frame in the first image frame sequence excluding the first image frame.

[0198] The backbone network is used to determine multiple color features (such as color information) corresponding to the first target image frame. Figure 10 The diagram illustrates color feature 1, color feature 2, ..., color feature m), and the weight parameters (e.g., ..., weights) corresponding to each color feature when representing target color information. Figure 10 The diagram illustrates weight parameter 1, weight parameter 2, ..., weight parameter m.

[0199] During training, the number of color features can be set according to the hardware capabilities of the mobile device. Understandably, the more color features there are, the more computational power the image denoising model will require.

[0200] A grid generation network is used to generate multiple grid sampling features that correspond one-to-one with the multiple color features based on the first color information and multiple color features. The multiple grid sampling features correspond to different image resolutions.

[0201] Specifically, each grid sampling network processes a color feature to generate a corresponding grid sampling feature. For example... Figure 10 As shown, grid sampling network 1 generates grid sampling feature 1 based on the first color information and color feature 1; grid sampling network 2 generates grid sampling feature 2 based on the first color information and color feature 2; ...; grid sampling network m generates grid sampling feature m based on the first color information and color feature m.

[0202] A grid decoding network is used to decode multiple grid sampling features based on multiple color features to obtain multiple grid decoding features corresponding to the multiple grid sampling features. The multiple grid decoding features correspond to the same image resolution.

[0203] Specifically, each grid decoding network decodes a grid sampling feature to generate a corresponding grid decoding feature. For example... Figure 10 As shown, mesh decoding network 1 decodes mesh sampling feature 1 based on color feature 1 to generate mesh decoding feature 1; mesh decoding network 2 decodes mesh sampling feature 2 based on color feature 2 to generate mesh decoding feature 2; ...; mesh decoding network m decodes mesh sampling feature m based on color feature m to generate mesh decoding feature m.

[0204] The fusion module is used to decode features based on multiple grids (such as...) Figure 10 The diagram illustrates mesh decoding feature 1, mesh decoding feature 2, ... mesh decoding feature m) and the weight parameters corresponding to multiple color features (e.g., ...). Figure 10 The weight parameters 1, 2, ..., m (shown in the diagram) are used to determine the denoised color information corresponding to the first target image frame.

[0205] Finally, the second target image frame corresponding to the first target image frame in the second image frame sequence is determined; and the loss value between the denoised color information corresponding to the first target image frame and the color information corresponding to the second target image frame is calculated. Specifically, the loss value between the color containing albedo corresponding to the denoised color information of the first target image frame and the color containing albedo corresponding to the color information of the second target image frame is calculated. Optionally, the loss value can be calculated using loss functions such as Smooth L1, L2, etc.

[0206] If the loss value is greater than the set threshold, the model parameters of the image denoising model are adjusted according to the loss value; if the loss value is less than or equal to the set threshold, the model is considered to have converged and training ends.

[0207] Alternatively, during model training, optimizers such as Adam can be used, with an initial learning rate set to lr = 1e-4 and eps = 1e-8, for a total of 100 epochs, with the learning rate decaying by 0.75 every 25 epochs. Optimizers such as SGD and RMSprop can also be used, with their corresponding decay strategies and epochs flexibly set.

[0208] This embodiment introduces the training process of the image denoising model from the perspective of model structure. Since the data processing process corresponding to the model training process is similar to that of the model usage process, this embodiment will not elaborate on the data processing process during model training. For the specific implementation process, please refer to the aforementioned image processing method embodiment.

[0209] This embodiment provides an image denoising model and a training method for the image denoising model. Since the network size of the image denoising model is small, the training speed is fast, and deploying the image denoising model only requires a small amount of computing power, so it can be deployed on mobile electronic devices to denoise images rendered by Monte Carlo path tracing.

[0210] Optionally, after the image denoising model is trained, it can be deployed on mobile electronic devices using tools such as Snapdragon Neural Processing Engine (SNPE), Quantum Neural Networks (QNN), TensorFlow Lite (TFLite), and coreML.

[0211] This application provides an electronic device in several embodiments, which may include a memory, a processor, and a communication interface. The memory stores executable code, which, when executed by the processor, enables the processor to at least implement the image processing method or the image denoising model training method provided in the foregoing embodiments. The structure of this electronic device can be referred to... Figure 1 The structure of the mobile electronic device 100 shown is illustrated.

[0212] In addition, this application embodiment also provides a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the image processing method provided in the foregoing embodiments.

[0213] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0218] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0219] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0220] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0221] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. An image processing method, characterized in that, include: Obtain the target image frame sequence rendered by Monte Carlo path tracing, and the color information corresponding to each image frame in the target image frame sequence; The target color information corresponding to the target image frame is determined based on the first color information corresponding to the previous image frame and the second color information corresponding to the target image frame. The target image frame is any image frame in the target image frame sequence except for the first image frame. Based on the target color information, determine multiple color features corresponding to the target image frame, and weight parameters corresponding to the multiple color features when representing the target color information; Based on the first color information and the plurality of color features, multiple grid sampling features corresponding one-to-one with the plurality of color features are generated, including: concatenating the albedo-removed color contained in the third color information with the plurality of color features through channels to obtain multiple channel concatenation results; performing grid sampling on the multiple channel concatenation results to obtain multiple grid sampling features; wherein, when performing grid sampling, the grid sampling parameters corresponding to each channel concatenation result are different; the third color information is obtained by mapping the first color information through a reprojection algorithm; the multiple grid sampling features correspond to different image resolutions; Decoding the multiple grid sampling features based on the multiple color features to obtain multiple grid decoding features corresponding to the multiple grid sampling features includes: performing upsampling processing on the multiple grid sampling features respectively, wherein the multiple grid sampling features after upsampling processing correspond to the same image resolution; calculating the second dot product results of the multiple grid sampling features after upsampling processing and the corresponding multiple color features respectively, to determine the multiple grid decoding features corresponding to the multiple grid sampling features after upsampling processing; wherein the multiple grid decoding features correspond to the same image resolution; Based on the weight parameters corresponding to the multiple grid decoding features and the multiple color features, the denoised color information corresponding to the target image frame is determined.

2. The method according to claim 1, characterized in that, The color information corresponding to each image frame includes: the albedo, pixel depth, pixel normal, and color after removing the albedo for each pixel in each image frame; wherein, the pixel depth is the depth of the pixel position relative to the virtual camera during the Monte Carlo path tracing rendering.

3. The method according to claim 2, characterized in that, The step of determining the target color information corresponding to the target image frame based on the first color information corresponding to the previous image frame and the second color information corresponding to the target image frame includes: Obtain the first position of the virtual camera when rendering the target image frame through the Monte Carlo path tracing in the previous image frame, and the second position of the virtual camera when rendering the target image frame through the Monte Carlo path tracing; Based on the first position and the second position, the first color information is mapped to the pixel coordinate system corresponding to the second color information to obtain the third color information corresponding to the first color information; The target color information corresponding to the target image frame is determined based on the third color information and the second color information.

4. The method according to claim 3, characterized in that, Determining the target color information corresponding to the target image frame based on the third color information and the second color information includes: Based on preset weighting coefficients, calculate the weighted average of the colors with reduced albedo contained in the third color information and the colors with reduced albedo contained in the second color information. Calculate the first dot product of the weighted average result and the albedo contained in the second color information; The first dot product result and the albedo, pixel depth, and pixel normal contained in the second color information are used as the target color information corresponding to the target image frame.

5. The method according to claim 1, characterized in that, The step of determining multiple color features corresponding to the target image frame based on the target color information, and the weight parameters corresponding to the multiple color features respectively when representing the target color information, includes: Extract the first color feature corresponding to the target color information; The number of channels of the first color feature is adjusted to the target number of channels to obtain the second color feature; The second color feature is sliced ​​according to the channel to determine multiple color features corresponding to the target image frame, and weight parameters corresponding to the multiple color features when representing the target color information; the number of multiple color features and the number of weight parameters are matched with the number of target channels.

6. The method according to claim 1, characterized in that, The grid sampling parameters include the downsampling coefficient and the channel adjustment coefficient.

7. The method according to claim 1, characterized in that, The step of determining the denoised color information corresponding to the target image frame based on the weight parameters corresponding to the plurality of grid decoding features and the plurality of color features includes: Based on the first correspondence between the plurality of grid decoding features and the plurality of color features, and the second correspondence between the plurality of color features and the weight parameters, the weight parameters corresponding to the plurality of grid decoding features are determined respectively. Based on the multiple grid decoding features and the weight parameters corresponding to the multiple grid decoding features, the weighted average result of the multiple grid decoding features is determined; Based on the weighted average result, the denoised color information corresponding to the target image frame is determined.

8. A method for training an image denoising model, characterized in that, include: Obtain a training sample set, which includes: a first image frame sequence and a second image frame sequence rendered by Monte Carlo path tracing for the same object, wherein the image noise in the first image frame sequence is greater than that in the second image frame sequence; The color information corresponding to each image frame in the first image frame sequence and the color information corresponding to each image frame in the second image frame sequence are input into the image denoising model, so that the image denoising model performs the following training operation: Based on the first color information corresponding to the previous image frame and the second color information corresponding to the first target image frame, the target color information corresponding to the first target image frame is determined. The first target image frame is any image frame in the first image frame sequence except for the first image frame. Based on the target color information, determine multiple color features corresponding to the first target image frame, and weight parameters corresponding to the multiple color features respectively when representing the target color information; Based on the first color information and the plurality of color features, multiple grid sampling features corresponding one-to-one with the plurality of color features are generated, including: concatenating the albedo-removed color contained in the third color information with the plurality of color features through channels to obtain multiple channel concatenation results; performing grid sampling on the multiple channel concatenation results to obtain multiple grid sampling features; wherein, when performing grid sampling, the grid sampling parameters corresponding to each channel concatenation result are different; the third color information is obtained by mapping the first color information through a reprojection algorithm; the multiple grid sampling features correspond to different image resolutions; Decoding the multiple grid sampling features based on the multiple color features to obtain multiple grid decoding features corresponding to the multiple grid sampling features includes: performing upsampling processing on the multiple grid sampling features respectively, wherein the multiple grid sampling features after upsampling processing correspond to the same image resolution; calculating the second dot product results of the multiple grid sampling features after upsampling processing and the corresponding multiple color features respectively, to determine the multiple grid decoding features corresponding to the multiple grid sampling features after upsampling processing; wherein the multiple grid decoding features correspond to the same image resolution; Based on the weight parameters corresponding to the multiple grid decoding features and the multiple color features, the denoised color information corresponding to the first target image frame is determined; Determine the second target image frame in the second image frame sequence that corresponds to the first target image frame; Calculate the loss value between the denoised color information corresponding to the first target image frame and the color information corresponding to the second target image frame; If the loss value is greater than a set threshold, the model parameters of the image denoising model are adjusted according to the loss value.

9. An electronic device, characterized in that, include: The device includes a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the image processing method as described in any one of claims 1 to 7, or the image denoising model training method as described in claim 8.

10. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the image processing method as described in any one of claims 1 to 7, or the image denoising model training method as described in claim 8.

Images