Model training methods, image processing methods and devices
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- VIVO MOBILE COMM HANGZHOU CO LTD
- Filing Date
- 2023-08-30
- Publication Date
- 2026-05-26
Smart Images
Figure CN117115593B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, specifically relating to a model training method, an image processing method, and an apparatus thereof. Background Technology
[0002] Unprocessed images directly output by image sensors often perform poorly in terms of noise, color, and distortion, resulting in poor image quality. Therefore, electronic devices with shooting capabilities usually include an image signal processor (ISP) to correct the unprocessed images directly output by the image sensor, resulting in corrected images with better visual quality.
[0003] An image signal processor (ESP) is a hardware unit that implements a set of fixed algorithms to process raw image data output from an image sensor and output a corresponding target format image. Specific operations performed by the ESP may include, but are not limited to, bad pixel correction, black level correction, noise reduction, lens shading correction, automatic white balance, depigmentation, color correction, gamma correction, sharpening, color space conversion, and encoding / decoding. For example, an ESP can convert raw image data (RAW image) output from an image sensor into a target format image (JPEG image) that is more visually appealing and easier to distribute.
[0004] However, traditional image signal processors require dedicated hardware chips, resulting in high deployment and application costs for traditional image processing methods. Summary of the Invention
[0005] The purpose of this application is to provide a model training method, an image processing method, and an apparatus thereof to solve the problem in the related art that processing raw format images requires the deployment of dedicated image signal processor hardware, resulting in high costs.
[0006] In a first aspect, embodiments of this application provide a model training method, the method comprising:
[0007] Obtain an unprocessed sample image and a corrected sample image corresponding to the unprocessed sample image;
[0008] The unprocessed sample image and the corrected sample image are feature-encoded to obtain the first image features;
[0009] The unprocessed sample image and the features of the first image are input into the first neural network model to obtain the second image output by the first neural network model.
[0010] The model loss is determined based on the corrected sample image and the second image, and the model parameters of the first neural network model are adjusted based on the model loss to obtain the second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
[0011] Secondly, embodiments of this application provide a model training method, the method comprising:
[0012] Acquire the unprocessed image;
[0013] The unprocessed image is input into the second neural network model to obtain the corrected image output by the second neural network model; wherein the second neural network model is trained by the model training method of the first aspect.
[0014] Thirdly, embodiments of this application provide a model training apparatus, the apparatus comprising:
[0015] The sample acquisition module is used to acquire unprocessed sample images and corrected sample images corresponding to the unprocessed sample images;
[0016] The feature encoding module is used to perform feature encoding on the unprocessed sample image and the corrected sample image to obtain the first image feature;
[0017] The sample input module is used to input the unprocessed sample image and the features of the first image into the first neural network model to obtain the second image output by the first neural network model.
[0018] The model training module is used to determine the model loss based on the corrected sample image and the second image, and to adjust the model parameters of the first neural network model based on the model loss to obtain the second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
[0019] Fourthly, embodiments of this application provide an image processing apparatus, the apparatus comprising:
[0020] The acquisition module is used to acquire unprocessed images;
[0021] The input module is used to input the unprocessed image into the second neural network model to obtain the corrected image output by the second neural network model; wherein the second neural network model is trained by the model training method of the first aspect.
[0022] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first or second aspect.
[0023] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first or second aspect.
[0024] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the first or second aspect.
[0025] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first or second aspect.
[0026] In this embodiment, an unprocessed sample image and a corresponding corrected sample image are first acquired. Feature encoding is then performed on the unprocessed and corrected sample images to obtain first image features. The unprocessed sample image and the first image features are then input into a first neural network model to obtain a second image output by the first neural network model. The model loss is determined based on the corrected sample image and the second image, and finally, the model parameters of the first neural network model are adjusted based on the model loss to obtain a second neural network model. The second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corresponding corrected sample image. By training the second neural network model, the image processing effects required for different scenarios and business applications can be met. The second neural network model provided in this solution can be trained based on sample images. By deploying the trained second neural network model on a device, unprocessed original images can be processed, reducing the hardware cost required for processing original format images. Attached Figure Description
[0027] Figure 1 This is a flowchart of a model training method provided in an embodiment of this application;
[0028] Figure 2 This is a schematic diagram of a Bayer array provided in an embodiment of this application;
[0029] Figure 3 This is a flowchart of an unprocessed sample image generation method provided in an embodiment of this application;
[0030] Figure 4 This is a flowchart of another model training method provided in the embodiments of this application;
[0031] Figure 5 This is a schematic diagram of an unprocessed raw image provided in an embodiment of this application;
[0032] Figure 6 This is a schematic diagram of an unprocessed sample image provided in an embodiment of this application;
[0033] Figure 7 This is a schematic diagram of an image rearrangement provided in an embodiment of this application;
[0034] Figure 8 This is a feature encoding flowchart provided in an embodiment of this application;
[0035] Figure 9 This is a schematic diagram of a first neural network model provided in an embodiment of this application;
[0036] Figure 10 This is a schematic diagram of a recursive attention block processing flow provided in an embodiment of this application;
[0037] Figure 11 This is a diagram illustrating an overall model training architecture provided in an embodiment of this application.
[0038] Figure 12 This is a flowchart of the steps of an image processing method provided in an embodiment of this application;
[0039] Figure 13 This is a block diagram of a model training device provided in an embodiment of this application;
[0040] Figure 14 This is a block diagram of an image processing apparatus provided in an embodiment of this application;
[0041] Figure 15 This is an electronic device provided in the embodiments of this application;
[0042] Figure 16 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0043] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0044] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or at least two. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0045] The model training method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0046] Reference Figure 1 , Figure 1 This application illustrates a flowchart of a model training method provided in an embodiment, applicable to electronic devices, such as... Figure 1 As shown, the specific steps include:
[0047] Step 101: Obtain the unprocessed sample image and the corrected sample image corresponding to the unprocessed sample image.
[0048] In electronic devices with shooting capabilities, an Image Signal Processor (ISP) is typically included. The ISP converts the raw image data output from the image sensor into a target image format that offers better visual appeal or is easier to distribute, such as converting a RAW image to a JPEG image. These electronic devices can include, but are not limited to, cameras, mobile phones, and tablets. The ISP directs hardware units that implement a series of fixed algorithms to process the raw image data output from the image sensor and output the corresponding target image format. Specific operations performed by the ISP may include, but are not limited to, bad pixel correction, black level correction, noise reduction, lens shading correction, automatic white balance, depigmentation, color correction, gamma correction, sharpening, color space conversion, and encoding / decoding. Image sensors can include, but are not limited to, CMOS sensors and CCD sensors, and their Bayer patterns can include, but are not limited to, any sensor array such as RGGB, RYYB, RGBW, and RGBIR. (See reference...) Figure 2 , Figure 2 A schematic diagram of a Bayer array provided in an embodiment of this application is shown.
[0049] Traditional image signal processors (IPCs) process images based on fixed algorithms and parameters, making them difficult to adjust quickly and easily to different scenarios and needs, resulting in poor flexibility. Furthermore, traditional IPCs may not achieve ideal image processing results when facing high image quality requirements. For example, in low-light or high-contrast environments, traditional IPCs may not be able to effectively remove noise, enhance sharpness, or expand dynamic range. This leads to a lower performance ceiling for traditional IPCs.
[0050] To address the shortcomings of traditional image signal processors, this application proposes a scheme that uses a second neural network model trained with neural network technology to replace the traditional image signal processor, thereby improving the flexibility of adjusting the image processing process and enhancing the image processing effect.
[0051] In this embodiment, the unprocessed sample image can represent raw image data directly output from the image sensor, typically without processing or compression. Unprocessed sample images retain the most image information and detail, but also occupy a larger storage space. The pixel arrangement of the unprocessed sample image can include, but is not limited to, RGGB, RYYB, RGBW, RGBIR, etc., and the image format can include, but is not limited to, RAW, DNG (Digital Negative), CR2 (Canon Raw Image File), NEF (Nikon Electronic Format), ARW (Sony Alpha Raw Image Format), etc.
[0052] A corrected sample image refers to image data processed from an unprocessed sample image, using the RGB color model to represent the color of each pixel. The RGB color model is an additive color model that generates various colors through different combinations of the three basic colors (red, green, and blue). Corrected sample images can be used for display, storage, or printing, and their formats can include, but are not limited to, JPEG, PNG, TIFF, and PSD.
[0053] In this embodiment, the unprocessed sample image can be obtained by capturing it using an electronic device that supports RAW format output, or by converting film or paper photographs to RAW format using a professional scanner or digitizer, or by directly acquiring the output signal of an image sensor; no specific limitation is made in this regard. The corrected sample image can be obtained by processing the RAW sample image using a conventional image signal processor.
[0054] Reference Figure 3 , Figure 3This application provides a flowchart for generating an unprocessed sample image according to an embodiment of the present application. Figure 3 As shown, the corrected sample image corresponding to the unprocessed sample image can be obtained by sequentially performing the following steps on the unprocessed sample image using a traditional image signal processor: re-mosaicing, bad pixel correction, black level correction, noise reduction, digital gain, lens shading correction, white balance correction, demosaicing, color correction, global gamma correction, RGB to YUV conversion, scaling, and sharpening.
[0055] Optionally, step 101 may include:
[0056] Sub-step 1011: Obtain the corrected sample image.
[0057] In this embodiment of the application, to further improve the training effect of the model, high-quality corrected sample images can be collected as training data. The sources of these corrected sample images may include, but are not limited to, networks, RGB image libraries, etc.
[0058] Sub-step 1012: Perform image degradation processing on the corrected sample image to obtain a degraded image.
[0059] Since it may be difficult to find corresponding original format image data for the high-quality corrected sample images collected, image degradation processing can be performed on the corrected sample images to simulate the characteristics and quality of the original format image data output from the image sensor, in order to generate corresponding unprocessed sample images. Degraded images refer to corrected sample images that have undergone a series of algorithmic processing, resulting in reduced sharpness, dynamic range, color reproduction, and other aspects.
[0060] In this embodiment, the aforementioned degradation processing may further include using algorithms such as Gaussian filtering, mean filtering, and median filtering to blur the corrected sample image, reducing sharpness and detail. Algorithms such as histogram equalization, gamma correction, and contrast enhancement may be used to adjust the brightness and contrast of the corrected sample image, reducing dynamic range and color reproduction. Algorithms such as Gaussian noise, salt-and-pepper noise, and Poisson noise may be used to add noise interference to the corrected sample image, reducing the signal-to-noise ratio and quality. Algorithms such as color space conversion, color correction, and white balance may be used to adjust the color of the corrected sample image, increasing color cast and distortion. Those skilled in the art can select the corresponding degradation processing procedure according to actual needs; this embodiment does not impose specific limitations.
[0061] Sub-step 1013: Generate the unprocessed sample image based on the degraded image.
[0062] In this embodiment of the application, the degraded image can be format converted to obtain the corresponding unprocessed sample image. During the format conversion process, the Bayer pattern can be used to rearrange the RGB pixels in the degraded image and fill the holes to obtain an unprocessed sample image that conforms to the pixel arrangement of the RAW format.
[0063] By performing degradation processing on the corrected sample images as described above, corresponding unprocessed sample images can be generated. This allows the model's training dataset to be constructed based on any RGB image, enabling the model training to use high-quality RGB images. This helps to improve the flexibility of building the training dataset and enhance the model training effect.
[0064] Step 102: Perform feature encoding on the unprocessed sample image and the corrected sample image to obtain the first image features.
[0065] In this embodiment, feature encoding can be performed on the unprocessed sample image and the corrected sample image to obtain the first image features. The function of feature encoding is to extract the main feature changes that occur in the image during the conversion from the unprocessed sample image to the corrected sample image, that is, to extract the quality difference features between the processed corrected sample image and the unprocessed sample image. Therefore, in the subsequent model training process, the first image features can guide the model to be closer to the traditional image signal processor in terms of image processing style, so that the trained second neural network model can inherit the processing effect of the image signal processor used to generate the corrected sample image.
[0066] In one implementation, the feature encoding described above can be achieved using a feature encoding network trained on a neural network. In another implementation, it can be obtained by comparing unprocessed sample images and corrected sample images. Specifically, image features of the unprocessed sample images and image features of the corrected sample images can be extracted, and the first image feature is determined by comparing the differences between the image features of the unprocessed sample images and the image features of the corrected sample images.
[0067] It should be noted that unprocessed sample images can include multiple consecutively exposed images output by the image sensor during a single shooting process. Multiple consecutively exposed images can carry richer image features, thereby extracting more accurate first image features. Here, consecutively exposed images can refer to multiple images taken of the same scene with different exposure parameters.
[0068] Step 103: Input the unprocessed sample image and the features of the first image into the first neural network model to obtain the second image output by the first neural network model.
[0069] In this embodiment, the first neural network model may include a convolutional neural network (CNN) model, such as ResNet, VGG, MobileNet, etc. Specific layers or modules can be added to these models to adapt to different image processing needs, such as noise reduction, super-resolution, color correction, etc. The first neural network model may also include attention-based neural network models, such as Transformer, BERT, etc. These models can capture long-range dependencies and global contextual information in the input data through attention mechanisms and dynamically assign different weights to different parts, thereby leveraging the characteristics of these models to achieve more refined and intelligent image processing effects. The first neural network model may also include generative adversarial networks (GANs) based neural network models, such as Pix2Pix, CycleGAN, StyleGAN, etc. These models can generate high-quality images through an adversarial learning process between a generator and a discriminator, thereby achieving more natural and high-quality image processing effects.
[0070] It should be noted that the input of the first neural network model includes the unprocessed sample image and the first image features. The first neural network model can learn how to process the first image features, process the unprocessed sample image, and output the corresponding second image.
[0071] Step 104: Determine the model loss based on the corrected sample image and the second image, and adjust the model parameters of the first neural network model based on the model loss to obtain the second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
[0072] In this embodiment, the model loss can be determined based on the corrected sample image and the second image. The first neural network model is then trained in a supervised manner based on the model loss to obtain the second neural network model. During supervised training based on the loss function, the parameters or weights in the model are continuously adjusted to minimize or optimize the value of the loss function. During training, algorithms such as stochastic gradient descent (SGD) and Adam can be used to calculate the gradient of the model loss with respect to the parameters or weights, and the parameters or weights are updated based on the direction and magnitude of the gradient.
[0073] The model loss mentioned above may include, but is not limited to, mean absolute error (MAE), mean squared error (MSE), cross-entropy (CE), binary cross-entropy (BCE), KL divergence (KLD), etc. Technicians can flexibly select appropriate loss functions to determine the model loss according to actual needs. The embodiments of this application do not impose specific limitations.
[0074] In summary, the model training method provided in this application includes: acquiring an unprocessed sample image and a corrected sample image corresponding to the unprocessed sample image; performing feature encoding on the unprocessed sample image and the corrected sample image to obtain a first image feature; inputting the unprocessed sample image and the first image feature into a first neural network model to obtain a second image output by the first neural network model; determining the model loss based on the corrected sample image and the second image, and adjusting the model parameters of the first neural network model based on the model loss to obtain a second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected sample image corresponding to the unprocessed sample image. By training the second neural network model, the image processing effects required under different scenarios and business conditions can be met. The second neural network model provided in this solution can be trained based on sample images. By deploying the trained second neural network model on a device, the unprocessed original image can be processed, which can reduce the hardware cost required for processing original format images to a certain extent.
[0075] Reference Figure 4 , Figure 4 This application provides an embodiment of another model training method flowchart, as shown below. Figure 4 As shown, the specific steps include:
[0076] Step 201: Obtain the unprocessed raw image output by the image sensor.
[0077] Step 202: Downsample the unprocessed original image to obtain the unprocessed sample image.
[0078] In this embodiment, since the unprocessed original image and the RGB original sample image typically have high resolution, in order to improve the training speed of the model, the unprocessed original image can be downsampled to obtain the unprocessed sample image. The downsampling method can include, but is not limited to, nearest neighbor downsampling, bilinear interpolation downsampling, bicubic interpolation downsampling, etc., and this embodiment does not impose specific limitations.
[0079] Step 203: Process the unprocessed sample image using an image processor to obtain the corrected sample image corresponding to the unprocessed sample image.
[0080] After obtaining the unprocessed sample images, the image processor is used to process the unprocessed sample images to directly obtain the corrected sample images corresponding to the unprocessed sample images. This allows the image processor to process the low-resolution unprocessed sample images, thereby reducing the power consumption of building the training set based on the image processor and improving the efficiency of building the training set.
[0081] Reference Figure 5 and Figure 6 , Figure 5 This illustration shows an unprocessed raw image provided in an embodiment of this application. Figure 6 This illustration shows an unprocessed sample image provided in an embodiment of this application, which can be processed by a downsampling unit. Figure 5 The unprocessed original image shown is subjected to 2x nearest neighbor downsampling to obtain the following result: Figure 6 The unprocessed sample image shown.
[0082] In this embodiment, an unprocessed original image and its corresponding RGB original sample image can also be obtained; wherein the unprocessed original image is obtained by processing the RGB original sample image output by the image sensor using an image processor. The unprocessed original image and the RGB original sample image can be downsampled separately to obtain an unprocessed sample image with a resolution lower than the unprocessed original image, and a corrected sample image with a resolution lower than the RGB original sample image. Then, the neural network model is trained based on the lower-resolution unprocessed sample image and the corrected sample image as a training set.
[0083] This process involves acquiring the raw, unprocessed image from the image sensor output; downsampling the raw, unprocessed image to obtain an unprocessed sample image; and then processing the unprocessed sample image using an image processor to obtain a corresponding corrected sample image. This reduces the amount of data in the training set, both in the unprocessed and corrected sample images. This not only improves the speed of model training but also allows the image processor to process lower-resolution unprocessed sample images, thereby reducing the power consumption of building the training set based on the image processor and improving the efficiency of training set construction.
[0084] Step 204: Generate the first multi-channel image data corresponding to the unprocessed sample image and the second multi-channel image data corresponding to the corrected sample image.
[0085] In this embodiment of the application, when inputting unprocessed sample images and corrected sample images into the model, the unprocessed sample images and corrected sample images can be stacked first to obtain stacked image data. By inputting the stacked image data into the model, the purpose of inputting unprocessed sample images and corrected sample images into the model can be achieved.
[0086] In this embodiment, pixel re-capture can be performed on the unprocessed sample image and the corrected sample image to generate first multi-channel image data corresponding to the unprocessed sample image and second multi-channel image data corresponding to the corrected sample image. Rearrangement is a method for converting a single-channel image into a multi-channel image. Its principle is to rearrange the pixels of each color channel into a new image according to certain rules based on the pixel arrangement of the single-channel image.
[0087] Reference Figure 7 , Figure 7 This illustration shows a schematic diagram of an image rearrangement provided in an embodiment of this application, such as... Figure 7 As shown, the unprocessed sample image 71 contains three color channels R, G, and B. The first multi-channel image data obtained after re-taking the unprocessed sample image 71 contains sub-images corresponding to each color channel, including sub-R channel sub-image 72, sub-G channel sub-image 73, and sub-B channel sub-image 74.
[0088] Step 2042: Stack the first multi-channel image data and the second multi-channel image data to obtain the stacked image data.
[0089] Step 205: Stack the first multi-channel image data and the second multi-channel image data and input them into the feature coding network to obtain the first image feature output by the feature coding network.
[0090] First multi-channel image data and second multi-channel image data are obtained, and then the first multi-channel image data and second multi-channel image data are stacked to obtain stacked image data. For example, if the first multi-channel image data contains 4 sub-images and the second multi-channel image data contains 3 sub-images, then the stacked image data obtained after stacking can contain 7 sub-images.
[0091] By generating first multi-channel image data corresponding to unprocessed sample images and second multi-channel image data corresponding to corrected sample images, and stacking the first multi-channel image data and the second multi-channel image data to obtain stacked image data, the color channels of unprocessed sample images and corrected sample images can be separated to form stacked image data. Model training using stacked image data can improve the refinement of the model input data, thereby improving the training effect of the model to a certain extent.
[0092] In this embodiment of the application, a first image feature can be obtained by extracting features from stacked image data through a feature coding network.
[0093] Reference Figure 8 , Figure 8 This application provides a feature encoding flowchart, as illustrated in the embodiment of the present application. Figure 8As shown, the unprocessed original image is rearranged to obtain the first multi-channel image data. After the first multi-channel image data and the corrected sample image are combined to generate stacked image data, the stacked image data is input into a feature encoding network to obtain the first image feature output by the feature encoding network. The feature encoding network can include a first encoding block, a second encoding block, and a third encoding block. Each encoding block can consist of N connected 3x3 convolutional layers with a stride of 1 and Rectified Linear Units (ReLUs), and one 3x3 convolutional layer with a stride of 2. If the stacked image data is represented as RGB∈R... H×W×6 Then, after stacking the image data and passing it through the first coding block, features can be obtained. Feature f1 can be processed into feature f1 after passing through the second coding block to obtain feature f1. Feature f2 can be processed by the third coding block to obtain the first image feature. Where H represents the width of the image, W represents the height of the image, and C1, C2, and C3 represent the number of feature layers. It should be noted that technicians can flexibly set the structure of the feature coding network according to actual needs; the feature coding network can also have any structure, and this application does not impose specific limitations.
[0094] By stacking unprocessed sample images and corrected sample images, stacked image data is obtained. The stacked image data is then input into a feature encoding network to obtain the first image feature output by the feature encoding network. This allows for joint feature extraction of unprocessed sample images and corrected sample images based on neural network technology. The extracted first image feature can accurately reflect the changes that occur during the image processing from unprocessed sample images to corrected sample images, thereby enabling the extracted first image feature to achieve better training results.
[0095] Step 206: Extract features from the multi-channel image data corresponding to the unprocessed sample image using the first recursive attention block unit to obtain the first sample image features.
[0096] In the embodiments of this application, the first neural network model may include an image rearrangement unit, a first recursive attention block unit, a feature fusion unit, a residual extraction layer, and a second recursive attention block unit.
[0097] Reference Figure 9 , Figure 9 This illustration shows a schematic diagram of a first neural network model provided in an embodiment of this application, such as... Figure 9As shown, the first neural network model may include an image rearrangement unit, a first recursive attention block unit, a feature fusion unit, a residual extraction layer, and a second recursive attention block unit. The first recursive attention block unit may consist of a first number of recursive attention blocks, and the second recursive attention block unit may consist of a second number of recursive attention blocks. A recursive attention block is a type of convolutional neural network that uses an attention mechanism. It can recursively analyze local information in an image and extract fine-grained information from the acquired local regions to obtain a more refined feature representation. Therefore, using recursive attention blocks allows attention to important details in the image, improving the accuracy of feature extraction.
[0098] Reference Figure 10 , Figure 10 This illustration shows a schematic diagram of a recursive attention block processing flow provided in an embodiment of this application, such as... Figure 10 As shown, the input of the recursive attention block is Input∈R H×W×C After batch normalization, the normalized result is processed through three parallel branches. Each branch extracts aggregated feature information through a network structure consisting of a 1x1 convolutional kernel (Conv1x1), a Rectified Linear Unit (ReLU), and a 3x3 convolutional kernel (Conv3x3). The results of each branch are then reshaped to obtain the mapped feature Q∈R. H×W×C Mapping features K∈R H×W×C and mapping features V∈R H×W×C The mapped features Q and K are then subjected to matrix multiplication and a softmax operation to obtain a result of size R. C×C The attention map is mapped, and the feature V is multiplied by the attention map. The result of the multiplication is transformed and then input into a Conv1x1 stride1 convolutional layer to obtain the attention residual feature. The input and the attention residual feature are added element by element. The result of the addition is processed by N cascaded network layers of Conv3x3+ReLU to obtain the output.
[0099] In one implementation, a first multi-channel image can be generated based on the unprocessed sample image. This first multi-channel image is then input into a first neural network model, where a first recurrent attention block unit processes the first multi-channel image to extract the features of the first sample image corresponding to the unprocessed sample image. In another implementation, such as... Figure 9As shown, the first neural network model may further include an image rearrangement unit. This unit can directly input the unprocessed sample image into the first neural network model, perform a space-to-depth operation on the unprocessed sample image to generate a corresponding first multi-channel image, and then the first recursive attention block unit extracts features from the first multi-channel image to obtain the first sample image features corresponding to the unprocessed sample image. The space-to-depth operation moves data from the spatial dimension to the depth dimension, thereby reducing spatial resolution and increasing the number of depth channels. The space-to-depth operation can reduce computational cost while extracting local features.
[0100] It should be noted that when the unprocessed sample image is obtained by downsampling the unprocessed original image, the first sample image features can be obtained by extracting features from the multi-channel image data corresponding to the unprocessed sample image through the first recursive attention block unit. Alternatively, the first sample image features can be obtained by extracting features from the multi-channel image data corresponding to the unprocessed original image through the first recursive attention block unit.
[0101] Step 207: The first image features and the first sample image features are fused by the feature fusion unit to obtain the second sample image features.
[0102] In this embodiment of the application, the first image features and the first sample image features can be fused by the feature fusion unit to obtain the second sample image features.
[0103] In one embodiment, the feature fusion unit can perform a feature concat operation on the first image features and the first sample image features, thereby fusing the first image features and the first sample image features into a second sample image feature. In another embodiment, such as... Figure 9 As shown, the feature fusion unit can also perform a resize operation on the first image features, aligning the first image features with the first sample image features to obtain aligned features with the same feature size (such as feature width, feature height, etc.) as the first sample image features. The feature fusion unit then performs a concat operation to fuse the aligned features and the first sample image features into second sample image features. The specific methods of the above feature alignment operation may include, but are not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., and are not specifically limited in this embodiment.
[0104] Step 208: Extract residuals from the features of the second sample image through the residual extraction layer to obtain residual features.
[0105] In the embodiments of this application, such as Figure 9 As shown, the first neural network model may further include a residual extraction layer, which is used to extract residuals from the features of the second sample image to obtain the feature residuals corresponding to the features of the second sample image. The residual extraction layer may be composed of a 3x3 convolutional kernel (Conv3x3) and a corrected linear unit (ReLU), or other structures may be used; this embodiment does not impose specific limitations.
[0106] Step 209: Input the second sample image features and the residual features into the second recursive attention block unit to obtain the second image.
[0107] In the embodiments of this application, such as Figure 9 As shown, the second sample image features obtained after feature concatenation and the residual features output by the residual extraction layer can be input into the second recursive attention block unit. The second recursive attention block unit further processes the input features to obtain the second image.
[0108] The first neural network model can learn how the image processor processes unprocessed sample images by using the input first image features, thereby improving the processing effect of the second neural network model trained based on the first neural network model. Furthermore, by introducing recursive attention blocks into the image processing model, the processing accuracy of image features is improved, which helps to further enhance the effect of processing unprocessed images based on the second neural network model.
[0109] Optionally, since the output of the second recursive attention block unit is an image feature, in order to obtain the corresponding second image based on the image feature, step 209 may include:
[0110] Sub-step 2091: Input the second sample image features and the residual features into the second recursive attention block unit to obtain the third sample image features output by the second recursive attention block unit.
[0111] Sub-step 2092: Determine the second multi-channel image data corresponding to the features of the third sample image.
[0112] In this embodiment of the application, the first neural network model can also generate corresponding second multi-channel image data based on the second multi-channel image data, and then synthesize each channel in the second multi-channel image data to obtain the second image.
[0113] Specifically, such as Figure 9As shown, the first neural network model can perform convolution processing on the third sample image features output by the second recurrent attention block unit through a convolutional layer to obtain the fourth sample image features. Then, a depth-to-space operation is performed on the fourth sample image features to obtain the second multi-channel image data. The depth-to-space operation moves the data from the depth dimension to the spatial dimension, thereby increasing the spatial resolution and reducing the number of depth channels. The convolutional layer can be composed of a 3x3 convolutional kernel (Conv3x3) and a rectified linear unit (ReLU), or other structures can be used; this embodiment does not impose specific limitations.
[0114] Sub-step 2093: Fuse the channels in the second multi-channel image data to obtain the second image.
[0115] In this embodiment, pixels from each channel of the second multi-channel image data can be filled into corresponding positions in the same image to obtain the second image. Alternatively, the images in the second multi-channel image data can be superimposed to obtain the second image; this embodiment does not specifically limit the method used.
[0116] By inputting the second sample image features and residual features into the second recursive attention block unit, the third sample image features output by the second recursive attention block unit are obtained. The second multi-channel image data corresponding to the third sample image features is determined. The various channels in the second multi-channel image data are fused to obtain the second image. Image data with multiple channels can be generated through image features, which helps to improve the accuracy of the pixels corresponding to the colors of each channel, thereby improving the accuracy of the final generated second image.
[0117] Step 210: Determine the model loss based on the corrected sample image and the second image, and adjust the model parameters of the first neural network model based on the model loss to obtain the second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
[0118] This step can be found in step 104, and the embodiments in this application are not specifically limited.
[0119] Reference Figure 11 , Figure 11 This application provides an embodiment of a model training architecture diagram, as shown below. Figure 11As shown, image sensor 111 outputs an unprocessed original image. This unprocessed original image is processed by downsampling unit 112 to obtain an unprocessed sample image. The unprocessed sample image is then processed by image processor 113 to obtain a corrected sample image. The unprocessed sample image and the corrected sample image are processed by feature encoding unit 114 to obtain a first image feature. The first image feature and the unprocessed original image are then input into a first neural network model 115 to obtain a second image output by the first neural network model 115. The second image can also be enhanced or used by post-processing unit 116. For example, the post-processing unit may include an image quality enhancement unit, which can perform image noise reduction and / or image super-resolution processing to improve the display effect of the second image. The post-processing unit may also include a target detection unit, which can be used to identify targets such as pedestrians, traffic lights, and traffic signs in autonomous driving tasks.
[0120] In summary, another model training method provided in this application includes: acquiring an unprocessed sample image and a corrected sample image corresponding to the unprocessed sample image; performing feature encoding on the unprocessed sample image and the corrected sample image to obtain a first image feature; inputting the unprocessed sample image and the first image feature into a first neural network model to obtain a second image output by the first neural network model; determining the model loss based on the corrected sample image and the second image, and adjusting the model parameters of the first neural network model based on the model loss to obtain a second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected sample image corresponding to the unprocessed sample image. By training the second neural network model, the image processing effects required under different scenarios and business conditions can be met. The second neural network model provided by this solution has the advantages of convenient adjustment and rapid deployment, which helps to improve the flexibility when adjusting the image processing method. It can also obtain a second neural network model with better image processing effect through training with a higher quality training set, which can surpass the image processing effect of traditional image processors in some aspects.
[0121] Reference Figure 12 , Figure 12 This application provides an embodiment of an image processing method, which includes the following steps:
[0122] Step 301: Obtain the unprocessed image.
[0123] Step 302: Input the unprocessed image into the second neural network model to obtain the corrected image output by the second neural network model.
[0124] The second neural network model is trained using the method provided in the above-described model training method embodiments, and will not be described again in this application embodiment.
[0125] In summary, the image processing method provided in this application can meet the image processing requirements of different scenarios and business scenarios by training a second neural network model. The second neural network model provided by this solution has the advantages of convenient adjustment and rapid deployment, which helps to improve the flexibility when adjusting the image processing method. Furthermore, a second neural network model with better image processing performance can be obtained by training a higher-quality training set, and in some aspects, it can surpass the image processing performance of traditional image processors.
[0126] Reference Figure 13 , Figure 13 This is a block diagram of a model training device provided in an embodiment of this application, such as... Figure 13 As shown, the model training device includes:
[0127] The sample acquisition module 1301 is used to acquire an unprocessed sample image and a corrected sample image corresponding to the unprocessed sample image;
[0128] Feature encoding module 1302 is used to perform feature encoding on the unprocessed sample image and the corrected sample image to obtain first image features;
[0129] The sample input module 1303 is used to input the unprocessed sample image and the features of the first image into the first neural network model to obtain the second image output by the first neural network model;
[0130] The model training module 1304 is used to determine the model loss based on the corrected sample image and the second image, and to adjust the model parameters of the first neural network model based on the model loss to obtain a second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
[0131] Optionally, the feature encoding module includes:
[0132] The combined image submodule is used to combine the unprocessed sample image and the corrected sample image to obtain combined image data;
[0133] The first image feature submodule is used to input the combined image data into the feature coding network to obtain the first image feature output by the feature coding network.
[0134] Optionally, the combined image submodule includes:
[0135] The first multi-channel image data submodule is used to generate first multi-channel image data corresponding to the unprocessed sample image and second multi-channel image data corresponding to the corrected sample image.
[0136] The stacking submodule is used to stack the first multi-channel image data and the second multi-channel image data to obtain the combined image data.
[0137] Optionally, the first neural network model includes an image rearrangement unit, a first recursive attention block unit, a feature fusion unit, a residual extraction layer, and a second recursive attention block unit, and the sample input module includes:
[0138] The first sample image feature submodule is used to extract features from the multi-channel image data corresponding to the unprocessed sample image through the first recursive attention block unit to obtain the first sample image features.
[0139] The second sample image feature submodule is used to fuse the first image features and the first sample image features through the feature fusion unit to obtain the second sample image features;
[0140] The residual feature submodule is used to extract residuals from the features of the second sample image through the residual extraction layer to obtain residual features;
[0141] The second image submodule is used to input the second sample image features and the residual features into the second recursive attention block unit to obtain the second image.
[0142] Optionally, the second image submodule includes:
[0143] The third sample image feature submodule is used to input the second sample image features and the residual features into the second recursive attention block unit to obtain the third sample image features output by the second recursive attention block unit.
[0144] The second multi-channel image submodule is used to determine the second multi-channel image data corresponding to the features of the third sample image;
[0145] The fusion channel submodule is used to fuse the various channels in the second multi-channel image data to obtain the second image.
[0146] Optionally, the sample acquisition module includes:
[0147] The unprocessed raw image submodule is used to acquire the unprocessed raw image output by the image sensor;
[0148] The unprocessed sample image submodule is used to downsample the unprocessed original image to obtain the unprocessed sample image.
[0149] The corrected sample image submodule is used to process the unprocessed sample image through an image processor to obtain the corrected sample image corresponding to the unprocessed sample image.
[0150] Optionally, the sample acquisition module includes:
[0151] The corrected sample image acquisition submodule is used to acquire the corrected sample image;
[0152] The degraded image submodule is used to perform image degradation processing on the corrected sample image to obtain a degraded image;
[0153] An unprocessed sample image generation submodule is used to generate the unprocessed sample image based on the degraded image.
[0154] In summary, the model training apparatus provided in this application includes: a sample acquisition module for acquiring unprocessed sample images and corresponding corrected sample images; a feature encoding module for encoding features in the unprocessed sample images and corrected sample images to obtain first image features; a sample input module for inputting the unprocessed sample images and the first image features into a first neural network model to obtain a second image output by the first neural network model; and a model training module for determining the model loss based on the corrected sample images and the second image, and adjusting the model parameters of the first neural network model based on the model loss to obtain a second neural network model. The second neural network model is used to perform image signal processing on the unprocessed sample images output by the image sensor to obtain the corresponding corrected images. By training the second neural network model, the image processing effects required in different scenarios and business scenarios can be met. The second neural network model provided in this solution has the advantages of convenient adjustment and rapid deployment, which helps to improve the flexibility when adjusting image processing methods. Furthermore, a second neural network model with better image processing effects can be obtained through training on a high-quality training set, surpassing the image processing effects of traditional image processors in some aspects.
[0155] Reference Figure 14 , Figure 14 This is a block diagram of an image processing apparatus provided in an embodiment of this application, such as... Figure 14 As shown, the image processing apparatus includes:
[0156] Acquisition module 1401 is used to acquire unprocessed images;
[0157] The input module 1402 is used to input the unprocessed image into the second neural network model to obtain the corrected image output by the second neural network model.
[0158] In summary, the image processing apparatus provided in this application can meet the image processing requirements of different scenarios and business scenarios by training a second neural network model. The second neural network model provided by this solution has the advantages of convenient adjustment and rapid deployment, which helps to improve the flexibility when adjusting the image processing method. Furthermore, a second neural network model with better image processing performance can be obtained by training a higher-quality training set, and in some aspects, it can surpass the image processing performance of traditional image processors.
[0159] The model training device and image processing device in the embodiments of this application can be electronic devices or components within electronic devices, such as integrated circuits or chips. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. The embodiments of this application do not impose specific limitations.
[0160] The model training device and image processing device in the embodiments of this application can be devices with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit them.
[0161] The model training device and image processing device provided in the embodiments of this application can achieve... Figures 1 to 7 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0162] Optionally, refer to Figure 15 , Figure 15 An electronic device provided in an embodiment of this application is shown, such as... Figure 15As shown, this application embodiment also provides an electronic device M00, including a processor M01 and a memory M02. The memory M02 stores a program or instructions that can run on the processor M01. When the program or instructions are executed by the processor M01, they implement the various steps of the above-described model training method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.
[0163] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.
[0164] Reference Figure 16 , Figure 16 A schematic diagram of the hardware structure of an electronic device according to an embodiment of this application is shown, such as... Figure 16 As shown:
[0165] The electronic device 1000 includes, but is not limited to, components such as: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, and processor 1010.
[0166] Those skilled in the art will understand that the electronic device 1000 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1010 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 16 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.
[0167] The processor 1010 is configured to acquire an unprocessed sample image and a corrected sample image corresponding to the unprocessed sample image; perform feature encoding on the unprocessed sample image and the corrected sample image to obtain a first image feature; input the unprocessed sample image and the first image feature into a first neural network model to obtain a second image output by the first neural network model; determine a model loss based on the corrected sample image and the second image, and adjust the model parameters of the first neural network model based on the model loss to obtain a second neural network model; wherein the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
[0168] The processor 1010 can also be used to acquire an unprocessed image; input the unprocessed image into a second neural network model to obtain a corrected image output by the second neural network model.
[0169] In summary, the electronic device provided in this application can meet the image processing requirements of different scenarios and business scenarios by training a second neural network model. The second neural network model provided by this solution has the advantages of convenient adjustment and rapid deployment, which helps to improve the flexibility when adjusting the image processing method. Furthermore, a second neural network model with better image processing performance can be obtained by training on a higher-quality training set, and in some aspects, it can surpass the image processing performance of traditional image processors.
[0170] It should be understood that, in this embodiment, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042. The GPU 10041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.
[0171] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in this embodiment includes, but is not limited to, these and any other suitable types of memory.
[0172] Processor 1010 may include one or at least two processing units; optionally, processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1010.
[0173] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above video shooting embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0174] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0175] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above video shooting embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0176] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0177] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video shooting embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0180] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A model training method, characterized in that, The method includes: Obtain an unprocessed sample image and a corrected sample image corresponding to the unprocessed sample image; The unprocessed sample image and the corrected sample image are feature-encoded to obtain the first image features; The unprocessed sample image and the features of the first image are input into the first neural network model to obtain the second image output by the first neural network model. The model loss is determined based on the corrected sample image and the second image, and the model parameters of the first neural network model are adjusted based on the model loss to obtain the second neural network model; The second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
2. The method according to claim 1, characterized in that, The step of performing feature encoding on the unprocessed sample image and the corrected sample image to obtain the first image feature includes: Generate first multi-channel image data corresponding to the unprocessed sample image, and second multi-channel image data corresponding to the corrected sample image; The first multi-channel image data and the second multi-channel image data are stacked and then input into a feature coding network to obtain the first image feature output by the feature coding network.
3. The method according to claim 1, characterized in that, The first neural network model includes an image rearrangement unit, a first recursive attention block unit, a feature fusion unit, a residual extraction layer, and a second recursive attention block unit. The step of inputting the unprocessed sample image and the features of the first image into the first neural network model to obtain the second image output by the first neural network model includes: The first sample image features are obtained by extracting features from the multi-channel image data corresponding to the unprocessed sample image through the first recursive attention block unit. The first image features and the first sample image features are fused by the feature fusion unit to obtain the second sample image features; The residual features are obtained by extracting residuals from the features of the second sample image through the residual extraction layer. The second sample image features and the residual features are input into the second recursive attention block unit to obtain the second image.
4. The method according to claim 3, characterized in that, The step of inputting the second sample image features and the residual features into the second recurrent attention block unit to obtain the second image includes: The second sample image features and the residual features are input into the second recursive attention block unit to obtain the third sample image features output by the second recursive attention block unit; Determine the second multi-channel image data corresponding to the features of the third sample image; The second image is obtained by fusing the various channels in the second multi-channel image data.
5. The method according to claim 1, characterized in that, The step of acquiring the unprocessed sample image and the corresponding corrected sample image includes: Obtain the corrected sample image; The corrected sample image is subjected to image degradation processing to obtain a degraded image; The unprocessed sample image is generated based on the degraded image.
6. An image processing method, characterized in that, The method includes: Acquire the unprocessed image; The unprocessed image is input into the second neural network model to obtain the corrected image output by the second neural network model; wherein the second neural network model is trained by the model training method according to any one of claims 1 to 5.
7. A model training device, characterized in that, The device includes: The sample acquisition module is used to acquire unprocessed sample images and corrected sample images corresponding to the unprocessed sample images; The feature encoding module is used to perform feature encoding on the unprocessed sample image and the corrected sample image to obtain the first image feature; The sample input module is used to input the unprocessed sample image and the features of the first image into the first neural network model to obtain the second image output by the first neural network model. The model training module is used to determine the model loss based on the corrected sample image and the second image, and to adjust the model parameters of the first neural network model based on the model loss to obtain the second neural network model; wherein, the second neural network model is used to perform image signal processing on the unprocessed sample image output by the image sensor to obtain the corrected image corresponding to the unprocessed sample image.
8. The apparatus according to claim 7, characterized in that, The feature encoding module includes: The first multi-channel image data submodule is used to generate first multi-channel image data corresponding to the unprocessed sample image and second multi-channel image data corresponding to the corrected sample image. The stacking submodule is used to stack the first multi-channel image data and the second multi-channel image data and input them into the feature coding network to obtain the first image feature output by the feature coding network.
9. The apparatus according to claim 7, characterized in that, The first neural network model includes an image rearrangement unit, a first recursive attention block unit, a feature fusion unit, a residual extraction layer, and a second recursive attention block unit. The sample input module includes: The first sample image feature submodule is used to extract features from the multi-channel image data corresponding to the unprocessed sample image through the first recursive attention block unit to obtain the first sample image features. The second sample image feature submodule is used to fuse the first image features and the first sample image features through the feature fusion unit to obtain the second sample image features; The residual feature submodule is used to extract residuals from the features of the second sample image through the residual extraction layer to obtain residual features; The second image submodule is used to input the second sample image features and the residual features into the second recursive attention block unit to obtain the second image.
10. The apparatus according to claim 9, characterized in that, The second image submodule includes: The third sample image feature submodule is used to input the second sample image features and the residual features into the second recursive attention block unit to obtain the third sample image features output by the second recursive attention block unit. The second multi-channel image submodule is used to determine the second multi-channel image data corresponding to the features of the third sample image; The fusion channel submodule is used to fuse the various channels in the second multi-channel image data to obtain the second image.
11. The apparatus according to claim 8, characterized in that, The sample acquisition module includes: The corrected sample image acquisition submodule is used to acquire the corrected sample image; The degraded image submodule is used to perform image degradation processing on the corrected sample image to obtain a degraded image; An unprocessed sample image generation submodule is used to generate the unprocessed sample image based on the degraded image.
12. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire unprocessed images; An input module is used to input the unprocessed image into a second neural network model to obtain a corrected image output by the second neural network model; wherein the second neural network model is trained by the model training method according to any one of claims 1 to 5.