Image processing method and device, and storage medium

Abstract

The invention discloses an image processing method and device, and a storage medium, and the method comprises the steps: obtaining a to-be-processed image at a first node in a preprocessing assembly line, and enabling the to-be-processed image to be an image collected by a first image collector; the to-be-processed image is input to an image enhancement model, a target image output by the image enhancement model is obtained, the image enhancement model is used for carrying out image enhancement on an image located at the first node of a preprocessing assembly line, and the image enhancement model is obtained based on training of at least two sampling images, the sampling image is an image collected by a second image collector and located at the first node of the preprocessing assembly line, the reference information of the to-be-processed image is the same as that of the sampling image, and the reference information is used for representing a key module of the corresponding image collector.

Description

Technical Field

[0001] The present application relates to terminal control technology, and in particular to an image processing method and device, and a storage medium. Background Art

[0002] Image enhancement algorithms are used in the following application scenarios: image deblurring, image super-resolution, and detail texture enhancement. In related technologies, image enhancement algorithms are supervised algorithms. Specifically, high-definition images taken with a SLR or open-source data are used as valid and correct (Ground True, GT) images. The GT images are downsampled to construct low-quality images. Training pairs including low-quality images and GT images are obtained. The image enhancement algorithm is supervised and trained based on the training pairs. However, high-definition images taken with a SLR have depth of field, and the difference between open-source data and actual data is too large. Therefore, the image enhancement effect is poor. Summary of the invention

[0003] The embodiments of the present application provide an image processing method and device, and a storage medium, which can improve the image enhancement effect.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides an image processing method, the method comprising:

[0006] Obtaining an image to be processed at a first node in the preprocessing pipeline, wherein the image to be processed is an image acquired by a first image acquirer;

[0007] The image to be processed is input into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by a second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

[0008] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0009] An obtaining unit, configured to obtain an image to be processed at a first node in the preprocessing pipeline, wherein the image to be processed is an image acquired by the first image acquirer;

[0010] A processing unit is configured to input the image to be processed into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by a second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

[0011] An embodiment of the present application provides an electronic device, including a processor, wherein the processor is configured to:

[0012] Obtaining an image to be processed at a first node in the preprocessing pipeline, wherein the image to be processed is an image acquired by a first image acquirer;

[0013] The image to be processed is input into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by a second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

[0014] An embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the above-mentioned image processing method are implemented.

[0015] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0016] The chip provided in the embodiment of the present application is used to implement the above-mentioned image processing method, and the chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned image processing method.

[0017] The image processing method, device, and storage medium provided in the embodiments of the present application use an image enhancement model generated based on a sampled image when performing image enhancement on an image to be processed, wherein a second image collector for acquiring the sampled image is identical to a key module of a first image collector for acquiring the image to be processed, so that the reference information of the image to be processed and the image to be processed are identical, and the position of the image to be processed in the preprocessing pipeline is identical to the position of the sampled image in the preprocessing pipeline, so that the image during training and the image during actual processing are actually captured images of the same source, thereby avoiding problems such as large data differences caused by the mismatch between high-quality shooting equipment and the first image collector, unclear image enhancement effects, and insufficient visual effects after image enhancement, thereby improving the image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0019] Figure 2 is an optional structural diagram of the image processing method provided in the embodiment of the present application;

[0020] Figure 3 It is an optional flowchart of the image enhancement model training process provided in the embodiment of the present application;

[0021] Figure 4 is an optional training schematic diagram of the first degradation model provided in an embodiment of the present application;

[0022] Figure 5 It is an optional flowchart of the image enhancement model training process provided in the embodiment of the present application;

[0023] Figure 6 is an optional training schematic diagram of the first degradation model provided in an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of the degenerate core effect provided in an embodiment of the present application;

[0025] Figure 8 It is an optional flowchart of the image enhancement model training process provided in the embodiment of the present application;

[0026] Fig. 9 It is an optional flowchart of the image processing method provided in the embodiment of the present application;

[0027] Fig.10 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0028] Fig.11is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0029] Fig.12 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0030] Fig.13 It is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] The embodiments of the present application can provide an image processing method, device and storage medium. In practical applications, the image processing method can be implemented by an electronic device, and each functional entity in the electronic device can be collaboratively implemented by hardware resources of a computer device (such as a terminal device, a network device, etc.), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).

[0033] Of course, the embodiments of the present application are not limited to being provided as methods and hardware, and there may be multiple implementation methods, such as being provided as a storage medium (storing instructions for executing the image processing method provided by the embodiments of the present application).

[0034] Below, various embodiments of the image processing method, device, and storage medium provided in the embodiments of the present application are described.

[0035] The image processing method provided in the embodiment of the present application can be as follows Figure 1 As shown, the method may include:

[0036] S101: An electronic device obtains an image to be processed at a first node in a preprocessing pipeline of a first image collector.

[0037] The electronic device may be provided with at least one image collector, wherein the first image collector is any image collector of at least one image collector in the electronic device.

[0038] The electronic device acquires an image through a first image collector. The image collector converts the captured light source signal into the original data of the digital signal, that is, the original image. The original image can be understood as the original file without preprocessing. The original image acquired by the image collector can be called a Bayer image or a RAW image. For a frame of the original image acquired by the image collector, it is preprocessed through a preprocessing pipeline. The preprocessing pipeline of the camera includes the following nodes: Bayer domain, RDB domain and YUV domain. The image in the Bayer domain is the original image, also called the Bayer domain image. The image in the RDB domain can be called the RDB domain image. The image in the YUV domain can be called the YUV domain image, wherein the Bayer domain can be understood as the input node of the preprocessing pipeline, and the YUV domain can be understood as the output node of the preprocessing pipeline. A frame of the Bayer domain image can be converted into an RGB domain image or a YUV domain image by format conversion. For the YUB domain image, it can also be directly obtained by converting the format of a frame of the RGB domain image. The RGB domain image includes pixel points distributed in an array, and each second pixel point includes a red sub-pixel point, a green sub-pixel point and a blue sub-pixel point. The YUV domain image includes a plurality of pixel points distributed in an array, and each pixel point includes a Y component, a U component, and a V component.

[0039] After the first image collector in the electronic device acquires the image, the electronic device can obtain the image to be processed at the first node of the preprocessing pipeline. The first node can be a Bayer domain, an RGB domain or a YUV domain.

[0040] In the embodiment of the present application, the electronic device may be provided with node indication information indicating the first node, the electronic device determines the first node according to the node indication information, and intercepts the image to be processed at the first node. For different image collectors in the electronic device, the corresponding first nodes may be the same or different.

[0041] S102. The electronic device inputs the image to be processed into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampled images, wherein the sampled images are images acquired by the second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampled images, and the reference information is used to characterize the key modules of the corresponding image collector.

[0042] In the embodiment of the present application, the image enhancement model is deployed on the first node of the preprocessing pipeline. After the electronic device obtains the image to be processed at the first node, it inputs the image to be processed into the image enhancement model to obtain the target image output by the image enhancement model. The target image is the image to be processed after image enhancement. In the embodiment of the present application, the image enhancement model has one or more image enhancement functions such as image deblurring, image super-resolution, and detail texture enhancement.

[0043] After the electronic device obtains the target image, it can continue to process the target image based on the preprocessing pipeline. If the first node is in the RGB domain, the electronic device can perform image preprocessing in the RDG domain such as gamma enhancement on the target image. If the first node is in the YUV domain, the electronic device can perform image preprocessing in the YUV domain such as noise reduction, edge enhancement, image cropping, and optical image stabilization on the target image.

[0044] In one example, if Figure 2 The Bayer image 201 acquired by the image collector is format-converted into an RDB domain image 202, and the RDB domain image 202 is input into an image enhancement model 203 to obtain a target image 204 output by the image enhancement model 203. The electronic device continues to perform RDB domain image processing on the target image 204 to obtain an image 205, and performs format conversion on the image 205 to obtain a YUV domain image 206.

[0045] The image enhancement model can be trained by a server, and the electronic device obtains it from the server through a network. The image enhancement model can also be trained by the electronic device itself.

[0046] After obtaining the image enhancement model, the electronic device deploys the image enhancement model at the first node of the preprocessing pipeline to enhance the image of the first node. Here, before the electronic device deploys the image enhancement model at the first node, it can quantify the image enhancement model according to the platform where the first node is located to convert the image enhancement model into a model file suitable for the first node. The platform in the embodiment of the present application can be understood as the platform of the real-time preprocessing pipeline, that is, the image processor. Among them, the image processor can be considered as a part of the image acquisition device, and the image processor also includes an image sensor for photosensitization, etc., wherein the key module of the image acquisition device belongs to the constituent module of the image sensor.

[0047] The image enhancement model for image enhancement in the embodiment of the present application is obtained by training at least two sampled images, and the sampled images are obtained by collecting the second image collector, and the sampled images are images of the first node of the corresponding preprocessing pipeline. Among them, different sampled images can be collected by the same or different second image collectors. The key modules such as the image collector and the filter of the first image collector and the second image collector are the same, so that the reference information of the image to be processed and the sampled image is the same. If the key module includes an image collector, the reference information includes: parameters such as noise distribution and resolution. If the key module includes a filter, the reference information includes: parameters such as spectrum and purity of RDB.

[0048] In the embodiment of the present application, the second image collector may be an image collector on the electronic device, or an image collector on another electronic device. If the second image collector is an image collector on the electronic device, the first image collector may also be a second image collector.

[0049] The sampled image in the embodiment of the present application is the image of the first node of the preprocessing pipeline acquired by the second image collector, thereby ensuring that the sampled image for training the image enhancement model and the image to be processed are homologous images with the same reference information, so that the image during training and the image during actual processing are actually taken images of the same source, avoiding the problems of large data differences, unclear image enhancement effects, and insufficient visual effects after image enhancement caused by the mismatch between high-quality shooting equipment and the first image collector, thereby improving the image enhancement effect.

[0050] In an embodiment of the present application, the image enhancement model can be trained by a server or by the electronic device itself.

[0051] In the scenario where the image enhancement model is trained by the server

[0052] In some embodiments, the electronic device also performs the following processing: receiving the image enhancement model and node indication information sent by the server, wherein the node indication information is used to indicate that the image enhancement model is used to process the image of the first node in the preprocessing pipeline.

[0053] Here, the image enhancement model is trained by a server, the server transmits the trained image enhancement model to the electronic device, and the electronic device receives the image enhancement model from the server.

[0054] In an embodiment of the present application, when the server transmits the image enhancement model to the electronic device, the server simultaneously transmits the node indication information to the electronic device to indicate to the electronic device which node in the preprocessing pipeline the image processed by the image enhancement model is. After receiving the node indication information, the electronic device deploys the image enhancement model at the corresponding node of the preprocessing pipeline based on the node indication information to perform image enhancement on the image of the corresponding node.

[0055] The server may actively send the image enhancement model to the electronic device, or may send the image enhancement model to the electronic device based on a request initiated by the electronic device.

[0056] In the image processing method provided in the embodiment of the present application, the image enhancement model can be trained by a server other than the electronic device, so that the electronic device only receives the image enhancement model from the service without performing the training process, thereby reducing the hardware processing capability requirements of the electronic device.

[0057] In some embodiments, the first image collector and the second image collector are the same image collector. In this case, the electronic device is also used to perform the following processing: acquiring at least two sampling images of the first image collector; sending the at least two sampling images to the server, and the at least two sampling images are used by the server to train the image enhancement model.

[0058] The electronic device may send at least two images acquired by the first image acquisition device as at least two sample images to the server. The electronic device may actively send the sample images to the server, or send the sample images to the server based on a request from the server. The multiple sample images for image enhancement model training may be sent by the electronic device at once or in batches.

[0059] The sampled image sent by the electronic device to the server is an image captured by the first image collector and located at the first node of the preprocessing flow, wherein, at the first node in the preprocessing pipeline, the electronic device copies the image of the node to obtain two images, sends one of the images to the server, and continues to preprocess the other image.

[0060] In the image processing method provided in the embodiment of the present application, when the electronic device needs to perform image enhancement on the image to be processed by the first image collector through the image enhancement model, the image collected by the first image collector can be sent to the server so that the server can train the image enhancement model through the image collected by the first image collector of the electronic device, thereby ensuring that the image to be processed and the sampled image are images collected by the same image collector, thereby improving the image enhancement effect.

[0061] In an embodiment of the present application, the sampled images used by the server for image enhancement model training may also come from other devices other than the electronic device where the first image collector is located.

[0062] In scenarios where the image enhancement model is trained by the electronic device itself

[0063] In the scenario where the image enhancement model is trained by the electronic device itself, the second image collector is an image collector in the electronic device.

[0064] In some embodiments, the first image collector and the second image collector are the same image collector, and the electronic device further implements the following processing:

[0065] Determining at least two sampled images of the first image collector;

[0066] Downsampling each of the at least two sampled images to obtain at least two degraded images;

[0067] The image enhancement model is trained based on the at least two sampled images and the at least two degraded images.

[0068] For each of the at least two sampled images, downsample the sampled image to obtain a degraded image after the sampled image is degraded, and the degraded image and the sampled image form an image pair. At least two pairs of images are obtained based on the at least two sampled images. The electronic device trains an image enhancement model based on the at least two pairs of images. The degradation processing performed on the degraded image relative to the corresponding sampled image includes: blur processing, motion blur processing, scaling processing, and other processing that reduces image quality.

[0069] For a sampled image, such as Figure 3 As shown, the sampled image 301 is downsampled to obtain a degraded image 302, and the sampled image 301 and the degraded image 302 are input into an initial image enhancement model 303, and the initial image enhancement model 303 is trained until the initial image enhancement model 303 converges. The converged initial image enhancement model is the trained image enhancement model 304.

[0070] The network structure of the image enhancement model may include one or more convolutional layers, one or more pooling layers, one or more fully connected layers, etc. The embodiment of the present application does not impose any limitation on the specific structure of the image enhancement model.

[0071] In an embodiment of the present application, the loss function used in training the image enhancement model is determined based on the sampled image and the degraded image.

[0072] In one example, the loss function L1 used in the training of the image enhancement model is shown in formula (1):

[0073] L1=(I1,I2)=||I1-I2||1+λΦ(I1,I2) Formula (1);

[0074] Where I1 is a degraded image, L2 is a sampled image, where a degraded image is also called a low quality (LQ) image, and a sampled image is also called a ground true (GT) image relative to a degraded image. Φ(I1, I2) is a function of the input and / or output of the initial enhancement model when I1 and L2 are input, and λ is a set adjustment coefficient, where the size of λ and the expression of Φ(I1, I2) can be adjusted according to the scene and style characteristics of the image enhancement.

[0075] In one example, an electronic device inputs multiple degraded images into an initial image enhancement model to obtain enhanced images corresponding to each degraded image output by the initial image enhancement model. The electronic device substitutes the enhanced image, the degraded image and the sampled image into a loss function to obtain a loss, adjusts the parameters of the initial image enhancement model based on the loss, and inputs multiple degraded images into the initial image enhancement model after parameter adjustment to obtain a new loss. When the obtained loss is greater than a loss threshold, the parameters of the initial image enhancement model continue to be adjusted, and multiple degraded images are input into the initial image enhancement model after parameter adjustment. When the obtained loss is less than or equal to the loss threshold, the initial image enhancement model is considered to have converged, and an image enhancement model is obtained.

[0076] In the image processing method provided in the embodiment of the present application, the electronic device trains an image enhancement model based on the image acquired by the first image acquirer, and applies the trained image enhancement model to image enhancement of the image to be processed acquired by the first image acquirer, thereby improving the image enhancement effect of the image enhancement model.

[0077] In some embodiments, downsampling each of the at least two sampled images to obtain at least two degraded images includes:

[0078] For each of the at least two sampled images, the following processing is performed: selecting a target degradation kernel from at least two degradation kernels; filtering the sampled image by using the target degradation kernel to obtain a degraded image corresponding to the sampled image.

[0079] In an embodiment of the present application, at least two degradation kernels are stored in the electronic device. When downsampling, the electronic device selects a degradation kernel from the at least two degradation kernels randomly or in sequence as a target degradation kernel, and downsamples the sampled image using the target degradation kernel.

[0080] In the embodiment of the present application, different degradation kernels are obtained by training different sampled images. The degradation kernel may include one or more of the following: blur kernel, motion blur kernel, scaling kernel, etc.

[0081] The degradation kernel in the embodiment of the present application can be characterized as a convolutional network including multiple convolutional layers, or it can be a feature map that characterizes the convolutional network.

[0082] The electronic device can train the degradation kernel by using at least two sampled images in a supervised or unsupervised manner. When a supervised manner is adopted, there is a reference degradation kernel, and supervised training is performed based on the reference degradation kernel. When an unsupervised manner is adopted, no degradation kernel needs to be known.

[0083] Training the degradation kernel in an unsupervised manner

[0084] In some embodiments, the electronic device further performs the following processing:

[0085] For each of the at least two sampled images, the following processing is performed to determine the at least two degradation cores:

[0086] The sampling image is input into a degradation generator to obtain a first degraded image; the sampling image and the first degraded image are respectively input into a degradation discriminator to obtain a thermal map of the sampling image and a thermal map of the first degraded image; based on a feature map of the reference degradation map and a thermal map of the reference degradation map, whether a first degradation model converges is judged, wherein the first degradation model includes the degradation generator and the degradation discriminator; if the first degradation model does not converge, the parameters of the first degradation model are updated, and the sampling image is continuously input into the degradation generator in the first degradation model to judge whether the first degradation model converges; if the first degradation model converges, the degradation kernel is determined based on the degradation generator.

[0087] In the embodiment of the present application, when the degradation kernel training is performed in an unsupervised manner, the degradation model used is Figure 4 The first degradation model shown is Figure 4 As shown, it includes: a degradation generator 401 and a degradation discriminator 402, wherein the degradation generator 401 is used to perform degradation processing on the sampled image 301, and its input is the sampled image, and its output is a first degraded image 403. The degradation discriminator 402 is used to determine the similarity between the first degraded image output by the degradation processor and the sampled image, and its input includes the sampled image 301 and the first degraded image 403, and its output is a heat map of the sampled image, i.e., a first heat map 404, and a heat map of the first degraded image, i.e., a second heat map 405. The first heat map 404 and the second heat map 405 are used to determine whether the sampled image 301 and the first degraded image 403 obey the same distribution.

[0088] In the embodiment of the present application, there is no limitation on the network structure of the degradation generator and the degradation discriminator.

[0089] In one example, the degradation generator includes five convolutional layers, and the parameters of the convolution kernel of each convolutional layer can be set according to requirements.

[0090] In one example, the degradation discriminator includes 6 convolutional layers, and each convolutional layer is also provided with a generalization layer and an activation function.

[0091] In the embodiment of the present application, the sampled image may be divided into blocks before being input into the degradation generator, and the image blocks obtained by the image division are input into the degradation generator.

[0092] In some embodiments, determining the degradation core based on the degradation generator includes: acquiring a reference image whose pixel values ​​are set values; inputting the reference image into the degradation generator to obtain a degradation feature map output by the degradation generator, and using the degradation feature map as the degradation core.

[0093] After completing the training of the first degradation model, the electronic device can obtain a reference image whose pixel values ​​are set values, input the reference image into the degradation generator, and obtain a degradation kernel corresponding to the degradation generator. At this time, the degradation kernel is a feature map that characterizes the convolutional network including multiple convolutional layers, namely, the degradation generator.

[0094] The value of the pixel of the reference image is a set value, which may be 1, thereby ensuring that no other elements are introduced when the degradation kernel is used.

[0095] In an embodiment of the present application, the degradation function of the network structure of the degradation generator is indirectly characterized by a feature graph. When the data amount of the feature graph is smaller than the data amount of the degradation generator, the complexity of the degradation processing of the feature graph is smaller than the complexity of the degradation processing of the degradation generator, which can effectively reduce the complexity of the training process of the image enhancement model.

[0096] The electronic device may also directly use the degradation generator as the degradation kernel. In this case, the degradation kernel is a convolutional network including multiple convolutional layers.

[0097] Training the degradation kernel in an unsupervised manner

[0098] For each of the at least two sampled images, the following processing is performed to determine the at least two degradation cores:

[0099] Based on a reference degradation kernel and the sampling image, the degradation deviation is determined, and an initial value of the reference degradation kernel is predicted based on a reference degradation map corresponding to the sampling image; when the degradation deviation is greater than a set degradation deviation, the reference degradation kernel is corrected based on the degradation deviation to obtain a new reference degradation kernel, a new sampling map is determined based on the reference degradation kernel and the reference degradation map, and new degradation deviations are continuously determined based on the reference degradation kernel and the sampling map until the determined degradation deviation is less than or equal to the set degradation deviation; and a reference degradation kernel corresponding to the degradation deviation that is less than or equal to the set degradation deviation is determined as the degradation kernel.

[0100] When the initial reference degradation kernel h0 is determined, the reference degradation kernel is iteratively corrected by the sampling image to obtain the final degradation kernel. The degradation deviation Δh0 is determined by the reference degradation kernel h0 and the sampling image. When Δh0 is greater than the set degradation deviation, a new reference degradation kernel h1 is determined based on Δh0 and h0, a new sampling image is determined based on h1 and the initial degradation image corresponding to the sampling image, and a new degradation deviation Δh1 is determined based on the new sampling image and the reference degradation kernel h1. If Δh1 is less than or equal to the set degradation deviation, the reference degradation kernel h1 is used as the final degradation kernel. Otherwise, a new reference degradation kernel h2 is determined based on Δh1 and h1, a new sampling image is determined based on h2 and the initial degradation image corresponding to the sampling image, and a new degradation deviation Δh2 is determined based on the new sampling image and the reference degradation kernel h2, until the determined degradation deviation is less than or equal to the set degradation deviation, and the reference degradation kernel corresponding to the degradation deviation is used as the final degradation kernel.

[0101] In the embodiment of the present application, the degradation model used for the degradation kernel training based on the unsupervised method is the second degradation model, and the second degradation model includes a predictor and a corrector, and the predictor is used to predict the initial reference degradation kernel h0 based on the initial degradation image corresponding to the sampled image. The predictor is used to predict the initial degradation kernel based on the initial degradation image of the sampled image, and the corrector is used to determine the degradation deviation based on the sampled image and the reference degradation kernel.

[0102] In the embodiment of the present application, the predictor may include 4 convolution layers and 1 global pooling layer. The 4 convolution layers use the reference degraded image, i.e., the low-resolution image, to predict the feature map of the reference degradation kernel. The global pooling layer performs average pooling on the feature map of the reference degradation kernel to obtain the reference degradation kernel. The convolution layer is used to extract blur features in the low-resolution image, and the global pooling can average the overall blur level spatially.

[0103] In an embodiment of the present application, the corrector performs feature transformation on the sampled image and the reference degradation kernel as the enhanced image to obtain their respective feature maps, then splices them together and uses multi-layer convolution to predict the feature map of the blur kernel correction amount, and finally uses global average pooling to obtain its vector representation Δh. The intermediate result of the enhanced image will produce over-smoothed or over-sharpened features due to the assumed deviation of the reference degradation kernel, and these deviation features are extracted by 5 convolutional layers. h is the PCA low-dimensional representation of the reference degradation kernel, so the correlation of each dimension should be as low as possible. Here, the internal correlation of h can be learned through 2 fully connected layers, and the corrector implicitly assumes that the internal correlation of h is the assumed deviation of the reference degradation kernel.

[0104] The image processing method provided in the embodiment of the present application can learn the degradation kernel based on the sampled image in a supervised or unsupervised manner, thereby ensuring the accuracy of the learned degradation kernel.

[0105] Next, the image processing method provided in the embodiment of the present application is further described.

[0106] Image enhancement algorithms are suitable for image deblurring, image super-resolution, and detail texture enhancement. In related technologies, image enhancement algorithms are supervised, specifically:

[0107] 1. Use a SLR to take high-definition pictures to make GT pictures, and use an artificially constructed downsampling method to downsample the GT pictures to obtain low-quality pictures. The GT pictures and low-quality pictures constitute a training pair, and the AI ​​algorithm is supervised based on the training pairs;

[0108] 2. Use open source data to make GT images, and use artificially constructed downsampling methods to downsample the GT images to obtain low-quality images. The GT images and low-quality images constitute training pairs, and supervised training of AI algorithms is performed based on the training pairs;

[0109] 3. Use pictures taken by SLR or open source pictures to make GT pictures. Use the downsampling method learned by AI to downsample the GT pictures and construct low-quality pictures. The GT pictures and low-quality pictures constitute training pairs, and the AI ​​algorithm is supervised based on the training pairs.

[0110] Application scenarios such as image deblurring, image super-resolution, and detail texture enhancement are highly dependent on high-quality data sets, but the acquisition of high-definition data has the following defects:

[0111] High-definition pictures taken with SLR cameras have a built-in depth of field. Natural images taken with SLR cameras will have many background blur effects. Although the foreground is very clear, there are still a lot of blurred areas on the picture. As a result, when doing supervised learning, a lot of blurred information is learned, affecting the learning quality.

[0112] There is a huge difference between open source data and actual data: In academia, a lot of open source data has been widely used. However, these open source data are very different from the data obtained by mobile phone ISP or taking pictures, and the sampling equipment of open source data varies greatly. If they are directly applied to the AI ​​model of the camera, the effect is often not ideal.

[0113] In response to the above problems, the embodiment of the present application constructs an unsupervised image enhancement algorithm that does not rely on data, which can get rid of the algorithm's dependence on data. And it can automatically learn its enhancement algorithm for different application scenarios, and the AI ​​model matures on its own. Its implementation process can be roughly divided into the following three steps:

[0114] The first step is to collect images in the target scene to obtain dump data, use the images of the target scene for downsampling learning, and learn to obtain the data degradation algorithm Deg;

[0115] The second step is to degrade the dump data using the data degradation algorithm learned in the first step to obtain degraded data.

[0116] The third step is to use the degraded data and dump data to form a training pair to train the AI ​​model, and apply real scene data for reasoning to obtain an unsupervised trained AI algorithm.

[0117] The image processing algorithm provided in the embodiment of the present application is as follows Figure 5 As shown, an image is captured by a camera 501 to obtain transferred data 502, where the transferred data is a sampled image, where the sampled image is the image of the first node of the preprocessing pipeline. The unsupervised degradation learning method 503 is learned through the transferred data 502 to obtain a downsampling model 504. The transferred data 502 is downsampled based on the downsampling model 504 to obtain degraded data, i.e., a degraded image 505. The transferred data 502 is used as a high-quality image and the degraded data 505 is used as a low-quality image to form an LQ-GT image pair. The image enhancement model 505 is trained based on the LQ-GT image pair to obtain a trained image enhancement model 507.

[0118] The trained image enhancement model 507 is deployed on the mobile phone to process the image 508 taken by the camera 501 to obtain an enhanced image 509. In the embodiment of the present application, the image enhancement model can be deployed on different nodes of image processing to process the corresponding image and obtain an image after image enhancement.

[0119] For example: the trained image enhancement model is directly deployed at the corresponding position of the mobile phone based on the node that collects data. Taking YUV image as an example, the data is collected at the node in the YUV domain, and the enhancement algorithm is deployed in the YUV domain, and then the captured picture is processed, and the result is the enhanced image.

[0120] First, a Bayer domain image of the current frame is provided, and then the Bayer domain image of the current frame is format-converted to obtain an RGB domain image and a YUV domain image, or first, a Bayer domain image of the current frame is provided, and then the Bayer domain image of the current frame is format-converted to obtain an RGB domain image, and finally the RGB domain image is format-converted to obtain a YUV domain image. The RGB domain image includes array-distributed pixels, and each second pixel includes a red sub-pixel, a green sub-pixel, and a blue sub-pixel. The YUV domain image includes a plurality of array-distributed pixels, and each pixel includes a Y component, a U component, and a V component.

[0121] In the embodiment of the present application, the image enhancement model training only uses the images taken by the mobile phone, so it is an unsupervised algorithm, and its learning direction is the high-quality image corresponding to the degradation method. According to research, the training time of this method is relatively long, and many texture details can be subjectively restored when applying reasoning, which greatly enhances image deblurring and image details and texture.

[0122] Unsupervised Learning Degradation Methods

[0123] The unsupervised learning degradation method in the embodiments of the present application includes but is not limited to using an adversarial generative model for degradation learning and using an iterative fuzzy kernel prediction method for degradation learning.

[0124] The process of using the adversarial generative model for degradation learning can be as follows Figure 6 As shown:

[0125] The picture 601 taken by the camera 501 is divided into blocks, and an image block 6011 is used as a GT image. The generator (G) 602 is used to perform feature recognition on the image block 6011 to obtain a blurred image 603, i.e., a fake label. The image block 6011 is used as a real label. The feature map 603 and the image block 6011 are input into a distribution discriminator 604, i.e., a degradation discriminator, and the output is a heat map 605 and a heat map 606. The heat map 605 and the heat map 606 are used to determine whether the blurred image 603 after convolution and the image block 6011 as the GT image obey the same distribution.

[0126] In one example, the structure of the generator 602 is as follows: Figure 6 As shown, it only includes five convolutional layers (1-st Conv, 2-ndConv, 3-rd Con v, 4-th Conv, 5-th Conv), and does not include an activation function. Therefore, the generator 602 does not change the resolution of the image block 6011. The function of the convolutional layer is to simulate the image degradation process. The image degradation process can be described as formula (2):

[0127] I_out =K*I _in Formula (2);

[0128] Among them, I _out This is the image after degradation. _in is the image before degradation, and K is the degradation kernel. Figure 6 As shown, convolution is performed using a convolution kernel that only includes the convolution layer and does not contain an activation function. Mathematically, this Figure 6 The degradation process shown can be described as formula (3):

[0129] I _out =K _1 *K _2 *K _3 *K _4 *K _5 *I _in Formula (3).

[0130] K _1 To K _5 They are the convolution kernels of the five convolution layers included in the generator 602.

[0131] K _1 To K _5 These five convolution kernels are combined to obtain formula (4):

[0132]

[0133] That is, the degradation kernel is learned. The reason for using five convolution kernels is to improve the fitting accuracy. Although the effect of a convolution kernel is similar in theory, the convergence is often insufficient, so using multiple kernels can make the learned degradation kernel more realistic. On the other hand, if more than five convolution layers are used, the effect of the learned degradation kernel is almost the same as that of using five convolution layers. Therefore, five convolution layers are used to learn degradation in the embodiment of the present application.

[0134] In one example, the structure of the distribution discriminator 603 is as follows: Figure 6 As shown, six convolutional layers are included, and different convolutional layers have their corresponding activation functions. The activation function may include: a combination of one or more of SpaceNorm, BatchNorm, and ReLU.

[0135] In one example, the parameters of the degradation generator are shown in Table 1, and the parameters of the distribution discriminator are shown in Table 2.

[0136] Table 1. Example of parameters for the degenerate core generator

[0137]

[0138] In Table 1, 1-st Conv (5x5, stride = 1, bias = False) means that the size of the convolution kernel of the first convolution layer is 5x5, the stride is 1, and there is no offset. The meanings of the parameters of other convolution layers are similar. For each layer in the degenerate generator, the size of the output image is the same as the size of the input image, so as to keep the size of the final output feature map and the size of the original image unchanged, so that there will be no blur caused by scaling. If there is a need for scaling, the scaling kernel can be added in the generator, and no scaling fitting is performed in the degenerate kernel.

[0139] Table 2. Example of parameters for distribution discriminator

[0140]

[0141]

[0142] In Table 2, 1-st Conv (7x7, stride = 1, bias = True, Act = SpaceNorm) indicates that the size of the convolution kernel of the first convolution layer is 7x7, the stride is 1, and the offset, Act = SpaceNorm indicates that the convolution layer has an activation function and the activation function is SpaceNorm, wherein Act = SpaceNorm + BatchNorm + ReLU indicates that the corresponding convolution layer has an activation function and the activation function includes SpaceNorm, BatchNorm, and ReLU. The meanings of the parameters of other convolution layers are similar.

[0143] In the embodiment of the present application, the loss function Loss(x) used in the training process can be expressed as formula (5):

[0144]

[0145] Among them, x~patch(I) represents the calculation of each image block, E represents the mathematical expectation, and E x～patch(I) () indicates that the variables in the brackets are averaged according to the image patch (I), D(x) indicates the output of the distribution discriminator D under the input x, G(x) indicates the output of the generator G under the input x, |D(G(x)| indicates the output of the distribution discriminator D under the input G(x), and R is expressed as formula (6):

[0146] R=αL norm +βL bound +γL sparse +δL center Formula (6);

[0147] Formula (6) has the constraints shown in formula (7):

[0148]

[0149] α, β, γ, δ are independent coefficients. In one example, the values ​​of α, β, γ, δ are 0.5, 0.5, 0.5, 0.5, respectively. k represents the element of the kernel after convolution by the generator. m represents the weight map (mask) weighted by a constant. (x0, y0) is the geometric center coordinate of the kernel. L norm is the regularization loss, which makes the cumulative sum of kernel elements equal to 1; L bound It is the boundary loss, which uses the mask to weight the kernel so that the kernel weight increases with the increase of the center distance. The closer to the boundary, the greater the penalty and the smaller the value. sparse is the sparse loss, which is used to ensure the sparsity of the kernel and prevent over-smoothing; L center It is the center loss, which is used to fix the kernel center position.

[0150] After the training is completed, the corresponding blur kernel can be obtained by inputting a 7x7 all-1 image, i.e., a GT image, into the degradation generator. The kernel effect obtained by learning can be shown as follows: Figure 7 As shown in Figure 2, different GT images obtain corresponding blur kernels.

[0151] Degradation learning using iterative blur kernel prediction

[0152] The predictor and corrector are constructed using a convolutional network. The predictor consists of 4 convolutional layers and a global pooling layer. The 4 convolutional layers use low-resolution images to predict blur kernel feature maps, and the global pooling layer averages the feature maps to obtain the blur kernel vector h. The convolutional layer is used to extract blur features in low-resolution images, while global pooling can average the overall blur level spatially. The corrector performs feature transformation on the enhanced image and blur kernel to obtain their respective feature maps, then splices them together and uses multi-layer convolution to predict the features of the blur kernel correction amount, such as, and finally uses global average pooling to obtain its vector representation Δh. The intermediate result of the enhanced image will produce over-smoothed or over-sharpened features due to the assumed deviation of the blur kernel. These deviation features are extracted by 5 convolutional layers. h is the PCA low-dimensional representation of the blur kernel, so the correlation of each dimension should be as low as possible. Therefore, two fully connected layers are used here to learn the internal correlation of h. The corrector implicitly believes that the internal correlation of h is the assumed deviation of the kernel. After splicing the above two deviations, they are converted into a global vector representation using the same method as the predictor.

[0153] The feature maps obtained by feature transformation of the enhanced image and blur kernel are concatenated to obtain A, and the correction amount B of the blur kernel is predicted using multi-layer convolution. The concatenation of A and B uses global average pooling to obtain their vector representation Δh. After concatenating h and Δh, feature transformation is used to convert them into a global vector representation.

[0154] The principle of the corrector is learning deviation. The intermediate result of the enhanced image will produce over-smoothed or over-sharp features due to the assumed deviation of the blur kernel. These deviation features are extracted by the corrector using 5 convolutional layers and then learned through 2 fully connected layers. Internal correlations are the assumed deviations of the kernel.

[0155] Downsampling and training an image augmentation model

[0156] In the process of training the degradation method, each image will generate a degradation kernel, which needs to be stored for subsequent downsampling operations. One GT image can get one degradation kernel, and N GT images can get N degradation kernels. When applying, randomly select a GT image or re-collect a GT image, and then randomly select a degradation kernel from the N degradation kernels and apply it to the GT image to obtain the LQ image corresponding to the GT image.

[0157] like Figure 8 As shown, the GT image 802 collected by the camera 501 is filtered using a degradation kernel 801 to obtain a blurred degraded image 803. The GT image 802 and the degraded image 803 are input to the initial image enhancement model 804. The initial image enhancement model 804 outputs I_GT corresponding to the GT image 802 and I_LQ corresponding to the degraded image 803. The loss function Loss is constructed using I_LQ and I_GT to train the image enhancement model 804, and the trained image enhancement model 805 is obtained. The construction of Loss can be adjusted and performed by task. In one example, Loss can be expressed as formula (8):

[0158] Loss(I LQ ,I GT )=||I LQ -I GT ||1+λΦ(I LQ ,I GT ) formula (8);

[0159] Among them, λΦ(I LQ ,I GT ) can be adjusted according to the scenario and style characteristics of the enhanced task.

[0160] Model Inference

[0161] Deploy the trained image enhancement model to the corresponding mobile device and directly process the captured image to achieve the purpose of image enhancement. Here, it is important to ensure that the Usecase of the application and the Usecase of the capture are the same.

[0162] like Fig. 9 As shown, the image to be processed 901 captured by the camera 501 is input into the image enhancement model 902 to obtain an output target image 903. The image enhancement model deployed at the first node can be a trained image enhancement model 904 quantified according to the image processing platform of the electronic device.

[0163] In the embodiment of the present application, the electronic device converts the image enhancement model into the model file of the corresponding platform according to the quantization and conversion method of the platform, keeps the node integrated by the camera unchanged, and integrates the model file into the camera, so that the image enhancement can be performed on the corresponding node. Among them, the platform can be understood as the chip used by the electronic device. The format requirements of platforms of different manufacturers or types are different. Thus, the model file adapted to the corresponding platform is obtained by conversion.

[0164] The image processing method provided in the embodiment of the present application uses data entirely from pictures actually taken by the camera for unsupervised learning and image enhancement, which makes the image enhancement algorithm no longer rely on high-definition data sets. In the actual training process of the image enhancement algorithm, it is very troublesome to obtain high-definition data sets. High-quality shooting equipment is likely to not match the mobile camera and there will be large data differences. The use of these high-definition data often results in problems such as the enhancement effect is not obvious and the visual effect after enhancement is not prominent enough. The use of this technical solution can get rid of this dependence and make the benefits of the enhanced algorithm after training obvious.

[0165] In addition, users can enhance data styles in degradation learning. In addition to ordinary blur, they can add data such as scaling and motion blur, so that the trained and stored degradation kernels can cover more scenes and adapt to enhancement tasks in more scenes, thereby providing better enhancement effects.

[0166] The present invention proposes an unsupervised image enhancement algorithm, which uses the data taken by the actual camera for degradation learning, and then applies it to the training of the enhancement algorithm, and the enhancement algorithm is trained and deployed. During the actual reasoning, the algorithm inputs the real pictures, so there is no data difference between the training process and the reasoning process. On the other hand, this process only requires real collected data for training. Unlike other algorithms, the present invention does not need to use SLR or other expensive sampling equipment to collect GT data, saving costs.

[0167] An electronic device according to an embodiment of the present application, such as Fig.10 As shown, the electronic device 1000 includes:

[0168] An obtaining unit 1001 is configured to obtain an image to be processed at a first node in a preprocessing pipeline, wherein the image to be processed is an image acquired by a first image acquirer;

[0169] The processing unit 1002 is configured to input the image to be processed into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by the second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

[0170] In some embodiments, the electronic device 1000 further includes:

[0171] A receiving unit is configured to receive the image enhancement model and node indication information sent by a server, wherein the node indication information is used to indicate that the image enhancement model is used to process the image of the first node in the preprocessing pipeline.

[0172] In some embodiments, the electronic device 1000 further includes:

[0173] A sending unit is configured to obtain at least two sampled images from the first image collector and send the at least two sampled images to the server, wherein the at least two sampled images are used by the server to train the image enhancement model; wherein the first image collector and the second image collector are the same image collector.

[0174] In some embodiments, the electronic device 1000 further includes:

[0175] A determination unit, configured to determine at least two sampled images of the first image collector; wherein the first image collector and the second image collector are the same image collector;

[0176] A down-sampling unit configured to down-sample each of the at least two sampled images to obtain at least two degraded images;

[0177] The first training unit is configured to train the image enhancement model based on the at least two sampled images and the at least two degraded images.

[0178] In some embodiments, the downsampling unit is configured to perform the following processing on each of the at least two sampled images:

[0179] selecting a target degenerate core from at least two degenerate cores;

[0180] The sampled image is filtered using the target degradation kernel to obtain a degraded image corresponding to the sampled image.

[0181] In some embodiments, the electronic device 1000 further includes:

[0182] The second training unit is configured to perform the following processing on each of the at least two sample images to determine the at least two degradation cores:

[0183] Inputting the sampled image into a degradation generator to obtain a first degraded image;

[0184] Inputting the sampled image and the first degraded image into a degradation discriminator respectively to obtain a heat map of the sampled image and a heat map of the first degraded image;

[0185] Determining whether a first degradation model converges based on a feature map of the reference degradation map and a heat map of the reference degradation map, wherein the first degradation model includes the degradation generator and the degradation discriminator;

[0186] If the first degradation model has not converged, updating the parameters of the first degradation model, and continuing to input the sampled image into the degradation generator in the first degradation model to determine whether the first degradation model has converged;

[0187] When the first degradation model converges, the degradation kernel is determined based on the degradation generator.

[0188] In some embodiments, the second training unit is further configured to:

[0189] Obtain a reference image whose pixel values ​​are set values;

[0190] The reference image is input into the degradation generator to obtain a degradation feature map output by the degradation generator, and the degradation feature map is used as the degradation kernel.

[0191] In some embodiments, the electronic device 1000 further includes:

[0192] The third training unit is configured to perform the following processing on each of the at least two sample images to determine the at least two degradation cores:

[0193] Determining the degradation deviation based on a reference degradation kernel and the sampled image, wherein an initial value of the reference degradation kernel is predicted based on a reference degradation map corresponding to the sampled image;

[0194] In the case where the degradation deviation is greater than a set degradation deviation, the reference degradation kernel is corrected based on the degradation deviation to obtain a new reference degradation kernel, a new sampling map is determined based on the reference degradation kernel and the reference degradation map, and a new degradation deviation is continuously determined based on the reference degradation kernel and the sampling map until the determined degradation deviation is less than or equal to the set degradation deviation;

[0195] A reference degradation core corresponding to the degradation deviation that is less than or equal to a set degradation deviation is determined as the degradation core.

[0196] In practical applications, the above-mentioned acquisition unit, processing unit, receiving and selecting unit, sending unit, determination unit, down-sampling unit, first training unit, second training unit, third training unit and other modules can be implemented by a processor located on an electronic device, specifically a combination of one or more of a central processing unit (CPU), a microprocessor (MPU), an application processor (AP), a digital signal processor (DSP), and a field programmable gate array (FPGA).

[0197] Those skilled in the art should understand that the relevant description of the above-mentioned electronic device in the embodiment of the present application can be understood by referring to the relevant description of the image processing method in the embodiment of the present application.

[0198] Fig.11 An optional implementation of the present application is provided as a structural diagram of an electronic device, such as Fig.11 As shown, the embodiment of the present application provides an electronic device 1100, including an electronic chip 1101, which can be implemented as an image processing method as described in one or more embodiments above. The electronic chip can be implemented as a processor of an electronic device or as a microprocessor of a fast charging module.

[0199] An embodiment of the present application provides an electronic device, Fig.12 Another optional implementation provided in the embodiment of the present application is a structural diagram of an electronic device, such as Fig.12 As shown, an embodiment of the present application provides an electronic device 1200, including:

[0200] A processor 1201 and a storage medium 1202 storing executable instructions of the processor 1201, wherein the storage medium 1202 relies on the processor 1201 to perform operations via a communication bus 1203, and when the instructions are executed by the processor 1201, the image processing method performed in one or more of the above embodiments is executed.

[0201] It should be noted that, in actual application, the various components in the terminal are coupled together via the communication bus 1203. It is understandable that the communication bus 1203 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 1203 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.12 Various buses are labeled as communication buses 1203.

[0202] An embodiment of the present application provides a computer storage medium, wherein the computer-readable storage medium is used to store a computer program, wherein the computer program enables a computer to execute the steps of the image processing method described in one or more of the above embodiments.

[0203] An embodiment of the present application provides a schematic structural diagram of an electronic device 1300. Fig.13 The electronic device 1300 shown includes a processor 1310. The processor 1310 is configured to:

[0204] For each image collector in at least one image collector, determine a configuration parameter set corresponding to the image collector, the configuration parameter set is generated by an image processor based on an image collected by the corresponding image collector, the configuration parameter set includes at least two configuration parameters, and different configuration parameters correspond to different frame images;

[0205] For each configuration parameter set, selecting a candidate configuration parameter from the configuration parameter set to determine at least one candidate configuration parameter, and storing the at least one candidate configuration parameter; the candidate configuration parameter is a valid configuration parameter in the configuration parameter set;

[0206] When the target image collector is started, the at least one candidate configuration parameter is read, and a target configuration parameter is selected from the at least one candidate configuration parameter read. The target configuration parameter is used by the image processor to process the target image data collected by the target image collector, and the target image collector is one of the at least one image collector.

[0207] In the embodiment of the present application, the processor 1310 can call and run a computer program from the memory to implement the image processing method in the embodiment of the present application.

[0208] Alternatively, if Fig.13 As shown, the electronic device 1300 may further include a memory 1320. The processor 1310 may call and run a computer program from the memory 1320 to implement the image processing method in the embodiment of the present application.

[0209] The memory 1320 may be a separate device independent of the processor 1310 , or may be integrated into the processor 1310 .

[0210] Alternatively, if Fig.13 As shown, the electronic device 1300 may further include a transceiver 1330, and the processor 1310 may control the transceiver 1330 to communicate with other devices, specifically, to receive signals sent by other devices. Here, the transceiver may include at least two antennas.

[0211] It is understandable that the transceiver includes multiple physical paths for receiving or transmitting signals, and the physical elements on one or more physical paths for transmitting signals constitute a transmitter, wherein the MPR values ​​of each physical path in the transmitter are independent.

[0212] Optionally, the electronic device 1300 can implement the corresponding processes implemented by the electronic device in each method of the embodiment of the present application, and for the sake of brevity, it will not be repeated here. It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Arra y, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0213] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0214] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate S DRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct RAM bus random access memory (Direct RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0215] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0216] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0217] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0218] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.

[0219] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0221] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0224] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0225] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining an image to be processed at a first node in the preprocessing pipeline, wherein the image to be processed is an image acquired by a first image acquirer; The image to be processed is input into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by a second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

2. The method according to claim 1, characterized in that: The method comprises: The image enhancement model and node indication information sent by the server are received, where the node indication information is used to indicate that the image enhancement model is used to process the image of the first node in the preprocessing pipeline.

3. The method according to claim 2, characterized in that The first image collector and the second image collector are the same image collector, and the method further includes: Acquire at least two sample images from the first image collector; The at least two sampled images are sent to the server, and the at least two sampled images are used by the server to train the image enhancement model.

4. The method according to claim 1, characterized in that: The first image collector and the second image collector are the same image collector, and the method further includes: Determining at least two sampled images of the first image collector; Downsampling each of the at least two sampled images to obtain at least two degraded images; The image enhancement model is trained based on the at least two sampled images and the at least two degraded images.

5. The method according to claim 4, characterized in that The step of downsampling each of the at least two sampled images to obtain at least two degraded images includes: For each of the at least two sample images, the following processing is performed: selecting a target degenerate core from at least two degenerate cores; The sampled image is filtered using the target degradation kernel to obtain a degraded image corresponding to the sampled image.

6. The method according to claim 5, characterized in that The method further comprises: For each of the at least two sampled images, the following processing is performed to determine the at least two degradation cores: Inputting the sampled image into a degradation generator to obtain a first degraded image; Inputting the sampled image and the first degraded image into a degradation discriminator respectively to obtain a heat map of the sampled image and a heat map of the first degraded image; Determining whether a first degradation model converges based on a feature map of the reference degradation map and a heat map of the reference degradation map, wherein the first degradation model includes the degradation generator and the degradation discriminator; If the first degradation model has not converged, updating the parameters of the first degradation model, and continuing to input the sampled image into the degradation generator in the first degradation model to determine whether the first degradation model has converged; When the first degradation model converges, the degradation kernel is determined based on the degradation generator.

7. The method according to claim 6, characterized in that The determining the degradation core based on the degradation generator comprises: Obtain a reference image whose pixel values ​​are set values; The reference image is input into the degradation generator to obtain a degradation feature map output by the degradation generator, and the degradation feature map is used as the degradation kernel.

8. The method according to claim 5, characterized in that The method further comprises: For each of the at least two sampled images, the following processing is performed to determine the at least two degradation cores: Determining the degradation deviation based on a reference degradation kernel and the sampled image, wherein an initial value of the reference degradation kernel is predicted based on a reference degradation map corresponding to the sampled image; In the case where the degradation deviation is greater than a set degradation deviation, the reference degradation kernel is corrected based on the degradation deviation to obtain a new reference degradation kernel, a new sampling map is determined based on the reference degradation kernel and the reference degradation map, and a new degradation deviation is continuously determined based on the reference degradation kernel and the sampling map until the determined degradation deviation is less than or equal to the set degradation deviation; A reference degradation core corresponding to the degradation deviation that is less than or equal to a set degradation deviation is determined as the degradation core.

9. An electronic device, characterized in that: The electronic device comprises: An obtaining unit, configured to obtain an image to be processed at a first node in the preprocessing pipeline, wherein the image to be processed is an image acquired by the first image acquirer; A processing unit is configured to input the image to be processed into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by a second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

10. An electronic device, comprising a processor, characterized in that: The processor is configured to: Obtaining an image to be processed at a first node in the preprocessing pipeline, wherein the image to be processed is an image acquired by a first image acquirer; The image to be processed is input into an image enhancement model to obtain a target image output by the image enhancement model, wherein the image enhancement model is used to perform image enhancement on the image located at the first node of the preprocessing pipeline, and the image enhancement model is trained based on at least two sampling images, wherein the sampling images are images acquired by a second image collector and located at the first node of the preprocessing pipeline, and the reference information of the image to be processed is the same as that of the sampling images, and the reference information is used to characterize the key modules of the corresponding image collector.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps in the image processing method according to any one of claims 1 to 8 are implemented.

12. A storage medium storing an executable program, characterized in that: When the executable program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

13. A chip, comprising a processor, characterized in that: The processor is configured to execute the image processing method according to any one of claims 1 to 8.

Images