Image processing method, image processing device and storage medium

Abstract

The present disclosure relates to an image processing method, an image processing device and a storage medium. The image processing method includes: obtaining a single-channel grayscale image; interpolating the green channel pixels of the single-channel grayscale image by an interpolation algorithm to obtain the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation of the single-channel grayscale image; using a pre-trained neural network model to adaptively interpolate and fuse the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation to obtain the green channel image of the single-channel grayscale image; using the green channel image as a reference template, using the interpolation algorithm to complete the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain a color image. Through the present disclosure, after the image is demosaiced, a color image with better image details and less pseudo-color is output.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device and a storage medium. Background Art

[0002] De-mosaicing technology, also known as demosaick algorithm, can expand a single-channel grayscale image from a single-channel grayscale image to a three-channel color image by completing the missing pixels in the red (R), green (G) and blue (B) channels, and according to certain strategies, adaptively enhance the clarity of the single-channel grayscale image and reduce the image noise.

[0003] With the development of camera technology, users have higher and higher requirements for images presented after taking pictures through camera devices. Therefore, how to better demosaic the single-channel grayscale image obtained after taking the image and output an image with better image details and less false colors is a new problem currently faced by demosaicing. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, an image processing device and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, the image processing method comprising: acquiring a single-channel grayscale image; interpolating green channel pixels of the single-channel grayscale image by an interpolation algorithm to obtain green channel horizontal pixel interpolation and green channel vertical pixel interpolation of the single-channel grayscale image; using a pre-trained neural network model to adaptively interpolate and fuse the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation to obtain a green channel image of the single-channel grayscale image; using the green channel image as a reference template, using an interpolation algorithm to complete the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain a color image.

[0006] In one example, the image processing method also includes: determining an image training set, the image training set including color training images; preprocessing the color training images in the image training set to obtain preprocessed color training images; determining the preprocessed color training images as comparison images of the output images of the neural network model; mosaic processing the comparison images to obtain a single-channel grayscale training image corresponding to the comparison images; and training the neural network model based on the single-channel grayscale training images and the comparison images.

[0007] In one example, the image training set includes one or more of: high-definition images acquired based on a public high-definition image set, high-frequency texture images, images taken by a designated camera module, and images taken by a single-lens reflex camera.

[0008] In one example, the comparison image is a comparison image obtained after brightness and noise are removed and edge enhancement is performed.

[0009] In one example, the pre-trained neural network model is a U-type neural network model.

[0010] According to a second aspect of an embodiment of the present disclosure, an image processing device is provided, the image processing device comprising: an acquisition unit, configured to acquire a single-channel grayscale image; a processing unit, configured to interpolate green channel pixels of the single-channel grayscale image through an interpolation algorithm to obtain green channel horizontal pixel interpolation and green channel vertical pixel interpolation of the single-channel grayscale image, and to adaptively interpolate and fuse the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation using a pre-trained neural network model to obtain a green channel image of the single-channel grayscale image, and to use the green channel image as a reference template to complete the red channel pixels and the blue channel pixels of the single-channel grayscale image using an interpolation algorithm to obtain a color image.

[0011] In one example, the image processing device also includes: a determination unit, configured to determine an image training set, the image training set including color training images; the processing unit is also configured to: preprocess the color training images in the image training set to obtain preprocessed color training images; determine the preprocessed color training images as a comparison image of the output image of the neural network model; mosaic process the comparison image to obtain a single-channel grayscale training image corresponding to the comparison image; and the training unit is configured to train the neural network model based on the single-channel grayscale training image and the comparison image.

[0012] In one example, the image training set includes one or more of: high-definition images acquired based on a public high-definition image set, high-frequency texture images, images taken by a designated camera module, and images taken by a single-lens reflex camera.

[0013] In one example, the comparison image is a comparison image obtained after brightness and noise are removed and edge enhancement is performed.

[0014] In one example, the pre-trained neural network model is a U-type neural network model.

[0015] According to a third aspect of the present disclosure, an image processing device is provided, the image processing device comprising: a memory configured to store instructions, and a processor configured to call the instructions to execute the image processing method in the first aspect or any one of the examples in the first aspect.

[0016] According to a fourth aspect of the present disclosure, a non-temporary computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they execute the image processing method in the aforementioned first aspect or any one of the examples in the first aspect.

[0017] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: before the interpolation and splicing of the horizontal pixels and the vertical pixels of the green channel of the single-channel grayscale image, the interpolation and the vertical pixels of the green channel of the single-channel grayscale image are corrected and fused by using the adaptive interpolation characteristics of the pre-trained neural network model, so that a high-accuracy green channel image can be obtained, and then, based on the high-accuracy green channel image, an interpolation algorithm is used to output a color image with accurate color. Through the present disclosure, it is possible to output a color image with better image details and less pseudo-color after de-mosaicing the image.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0021] Figure 2 It is a schematic diagram of the Unet network structure using the present disclosure according to an exemplary embodiment.

[0022] Figure 3 It is a flow chart of training a Unet model according to an exemplary embodiment.

[0023] Figure 4 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0024] Figure 5 It is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0026] The technical solution of the exemplary embodiments of the present disclosure can be applied to application scenarios in which shooting is performed using a terminal including a shooting function. In the exemplary embodiments described below, the terminal is sometimes also referred to as an intelligent terminal device, wherein the terminal may be a mobile terminal, or may be referred to as a user equipment (UE), a mobile station (MS), etc. A terminal is a device that provides voice and / or data connection to a user, or a chip disposed in the device, such as a handheld device with a wireless connection function, a vehicle-mounted device, etc. For example, examples of terminals may include: mobile phones, tablet computers, laptop computers, PDAs, mobile Internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in unmanned driving, wireless terminals in remote surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc.

[0027] At present, the mainstream demosaick algorithm uses interpolation algorithm to implement demosaick. The core idea is to use the interpolation result of G channel as the benchmark and complement the R and B channels by predicting the difference between R, B channels and G channel. However, with the development of camera technology, how to better demosaic and output images with better image details and less false colors is a new problem facing demosaicing.

[0028] Furthermore, in order to further improve the effect of demosaick using interpolation algorithms, the most popular deep convolutional neural network in artificial intelligence (AI) has begun to be used in demosaick. However, while convolutional neural networks can improve demosaicking performance, they have a complex structure, too many calling parameters, and slow running speed. This is the biggest problem of AI-demosaick now.

[0029] Based on this, the present disclosure provides an image processing method that can combine an interpolation algorithm and a convolutional neural network to demosaic an image, thereby improving the image demosaicing performance while ensuring that the image demosaicing processing speed is relatively fast and meets the commercial landing requirements.

[0030] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method includes the following steps.

[0031] In step S11, a single-channel grayscale image is acquired.

[0032] In the present disclosure, a single-channel grayscale image may be a single-channel grayscale image captured by an image sensor in a camera device when the camera device is used for shooting. For example, the single-channel grayscale image may be a single-channel grayscale image arranged in a Bayer array.

[0033] The camera device may be a digital camera, a single-lens reflex camera, a camera device included in a smart phone, or the like.

[0034] In step S12, the green channel pixels of the single-channel grayscale image are interpolated by an interpolation algorithm to obtain the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation of the single-channel grayscale image.

[0035] In the present disclosure, the interpolation algorithm may be, for example, a bilinear interpolation algorithm, a bicubic interpolation algorithm, a residual interpolation (RI) interpolation algorithm, etc. The residual interpolation algorithm may be, for example, a minimum Laplace residual interpolation algorithm (MLRI) or an adaptive residual interpolation algorithm (ARI).

[0036] In the present disclosure, for example, the green channel pixel interpolation of a single-channel grayscale image may be performed in the following manner:

[0037] The single-channel grayscale image is split into R channel pixel image, G channel pixel image and B channel pixel image. Green interpolation correction (GIC) is performed on the G channel pixel image of the single-channel grayscale image, that is, the error between the green and red color components and the error between the green and blue color components of the G channel pixel image are corrected, and then the green channel pixels corrected by GIC are interpolated using an interpolation algorithm to obtain the horizontal pixel interpolation of the green channel and the vertical pixel interpolation of the green channel of the single-channel grayscale image.

[0038] In step S13, the pre-trained neural network model is used to perform adaptive interpolation fusion on the horizontal pixel interpolation and the vertical pixel interpolation of the green channel of the single-channel grayscale image to obtain the green channel image of the single-channel grayscale image.

[0039] At present, when an interpolation algorithm is used to demosaic an image, the obtained horizontal pixel interpolation and vertical pixel interpolation of the G channel are directly spliced ​​to obtain a green channel image.

[0040] After the G channel pixels are interpolated by the interpolation algorithm, the horizontal and vertical pixel interpolations of the G channel obtained have errors, and the color of the green channel image obtained by directly splicing the horizontal and vertical pixel interpolations of the G channel is inaccurate. Then, when the inaccurate green channel image is used as a reference template and the red channel pixels and blue channel pixels of the single-channel grayscale image are complemented by the interpolation algorithm, the color of the red channel image and the blue channel image obtained are also inaccurate.

[0041] Therefore, the present invention can utilize the adaptive interpolation characteristics of a pre-trained neural network model to correct the interpolation of horizontal pixels and vertical pixels of the green channel of a single-channel grayscale image before splicing them, and then fuse the corrected interpolation of horizontal pixels and vertical pixels of the green channel to obtain a green channel image with high accuracy.

[0042] The pre-trained neural network model may be, for example, a U-type neural network model (Unet) or a demosaicing convolutional neural network (DmCNN).

[0043] In the present disclosure, a pre-trained neural network model Unet is taken as an example to illustrate how the pre-trained neural network model is used to adaptively interpolate and fuse the horizontal pixels and vertical pixels of the green channel of a single-channel grayscale image to obtain a green channel image of the single-channel grayscale image.

[0044] Since the Unet model can adaptively correct the horizontal pixel channel and vertical pixel channel of the interpolated image, after using the Unet model to correct the errors of the horizontal pixel interpolation and vertical pixel interpolation of the G channel, the horizontal pixel interpolation and vertical pixel interpolation of the G channel after the error correction are fused to obtain the accurate green channel image of the single-channel grayscale image.

[0045] The interpolation algorithm and the Unet model are used to demosaic a single-channel grayscale image. For example, it can be applied to the digital image demosaicing technology developed for the ISP chip in the terminal to restore the single-channel grayscale image in the Bayer array pattern into a complete red, blue and green three-channel color image.

[0046] Figure 2 This is a schematic diagram of the Unet network structure using the present disclosure. Figure 2 In the Unet model, since the Unet model only needs to fuse the horizontal pixel interpolation and the vertical pixel interpolation of the green channel, the number of convolutional boxes in the Unet model only needs to be 6. After the image is input into the Unet model, it passes through the encoder and then the decoder. The encoder (decoder) part includes three layers of convolutional boxes (conv Block), and then the image passes through the encoder and the decoder part, which also includes three layers of convolutional boxes (conv Block). There can be a shortcut connection between the encoder and the decoder to increase the learning performance of the Unet model for image features.

[0047] Furthermore, in order to enhance the invariance of image scale, up and down sampling is added between every two convolution boxes of the Unet model, that is, when the horizontal pixel interpolation and the vertical pixel interpolation of the green channel are fused through the Unet model disclosed in the present invention, an enlarged single-channel grayscale image after the single-channel grayscale image is enlarged and a thumbnail single-channel grayscale image after the single-channel grayscale image is reduced is added between every two convolution boxes, so that when the horizontal pixel interpolation and the vertical pixel interpolation of the green channel are fused, multi-scale image fusion can be achieved.

[0048] In step S14, the green channel image is used as a reference template, and an interpolation algorithm is used to complement the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain a color image.

[0049] In the present disclosure, the interpolation of the horizontal pixels and the interpolation of the vertical pixels of the green channel of a single-channel grayscale image are fused through a pre-trained neural network model to obtain an accurate green channel image. Then, based on the accurate green channel image, a residual interpolation algorithm is used to complete the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain accurate red channel images and blue channel images, which are then output as a color image.

[0050] For example, using the residual interpolation algorithm to complete the red channel pixels of a single-channel grayscale image can be achieved as follows:

[0051] The green channel image fused by the Unet model is used as the reference image, and guided upsampling is performed to obtain the initialized red channel image. The initialized red channel image is then operated on the red channel pixels of the single-channel grayscale image to obtain the minimum Laplace residual. Finally, the upsampled residual result is added back to the red channel pixels of the single-channel grayscale image to obtain the red channel image of the single-channel grayscale image.

[0052] Similarly, the green channel image fused by the Unet model is used as the reference image, and the blue channel pixels are complemented to obtain the blue channel image of the single-channel grayscale image. The green channel image of the single-channel grayscale image, the blue channel image of the single-channel grayscale image, and the red channel image of the single-channel grayscale image are combined to obtain the output color image.

[0053] In an exemplary embodiment of the present disclosure, before the interpolation of horizontal pixels and vertical pixels of the green channel of a single-channel grayscale image are spliced, the interpolation of horizontal pixels and vertical pixels of the green channel of the single-channel grayscale image are corrected and fused using the adaptive interpolation characteristics of a pre-trained neural network model, so that a green channel image with high accuracy can be obtained, and then a color image with accurate color can be output based on the green channel image with high accuracy using a residual interpolation algorithm. Through the present disclosure, it is possible to output a color image with better image details and less pseudo-color after de-mosaicing the image.

[0054] In the present disclosure, before using an interpolation algorithm and a pre-trained neural network model to demosaic a single-channel grayscale image to obtain a color image, a neural network model may be trained.

[0055] The present disclosure below uses the Unet model as an example of a neural network model to illustrate the training of a de-mosaicing neural network.

[0056] Figure 3 is a flow chart of training a Unet model according to an exemplary embodiment. Figure 3 As shown, the method for training the Unet model includes the following steps.

[0057] In step S21, an image training set is determined.

[0058] Since the training set plays a vital role in training the neural network model, in order to make the demosaicing effect of the trained Unet model better, the image training set can be obtained specifically according to the camera module to which the demosaicing needs to be applied. For example, according to the parameters of the camera module installed in terminal A, the image training set suitable for the camera module of terminal A can be determined specifically. Then, during training, it is more targeted, so that the performance of the training Unet model will be better.

[0059] The image training set includes color training images, which can be based on one or more of high-definition images obtained from a public high-definition image set, high-frequency texture images, images taken by a designated camera module, and images taken by a single-lens reflex camera.

[0060] High-definition images obtained from public high-definition image sets can be used to initialize the parameters of the Unet model and fix the parameters of the Unet model within a certain range. Images with designed high-frequency textures can be used to improve the texture restoration ability of the Unet model for images. Images taken by specifying the camera module to which de-mosaicing needs to be applied can be used to improve the robustness of the Unet model. Images taken by a single-lens reflex camera can be used to improve the image quality enhancement capabilities of the Unet model, such as improving the noise reduction capability of the image and improving the texture clarity of the image.

[0061] Therefore, when the Unet model is trained by the above-mentioned image training set, since the image training set is specially designed for application to a specified camera module, training the Unet model according to the targeted image training set can make the trained Unet model more targeted and directed, and when the trained Unet model is deployed to the digital image processing platform that applies the specified camera module, the image can be processed faster. In addition, based on the de-mosaicing of the image in the specified camera module, the color image obtained is more realistic and the noise intensity is weakened, and the color image output by the Unet model has higher clarity, and the image output can still have better color reproduction in extremely dark and extremely bright extreme lighting environments.

[0062] In step S22, a comparison graph of the single-channel grayscale training image and the Unet model output image is obtained according to the image training set.

[0063] In one embodiment, the present disclosure may preprocess the color training images in the image training set to obtain the preprocessed color training images, wherein the preprocessing may include at least one of de-gamma and de-white balance of the color training images.

[0064] In order to improve the training speed of the Unet model, the present disclosure can crop the preprocessed color training image into small images, for example, the preprocessed color training image can be cropped into a small image with a pixel size of 30*30, and a comparison chart of the Unet model output image can be obtained. In addition, if the resolution of the preprocessed color training image exceeds the maximum value required by the training specified camera module, such as images taken by a single-lens reflex camera or high-definition images obtained based on a public high-definition image set, the image can be downsampled to a suitable size before cropping. For example, the image is downsampled to an image with a pixel size of 3000*4000.

[0065] After obtaining the comparison image of the Unet model output image, the present invention performs mosaic processing on the comparison image to obtain a single-channel grayscale training image corresponding to the comparison image and input into the U-type neural network.

[0066] In one embodiment, in order to enable the trained Unet model to enhance the image and reduce noise while demassage, the present disclosure may also remove brightness and noise from the contrast image and perform edge enhancement processing to obtain a processed contrast image. Based on the processed contrast image and the single-channel grayscale training image, the Unet model is trained.

[0067] In step S23, a U-type neural network model is trained based on the single-channel grayscale training image and the comparison image.

[0068] The single-channel grayscale training image and the comparison image are input into the Unet model, and the color image of the single-channel grayscale training image is predicted by the Unet model to obtain the color prediction image corresponding to the single-channel grayscale training image. The error between the color prediction image and the comparison image is calculated according to the loss function, and the parameters of the Unet model are adjusted according to the calculated error until the error calculated by the loss function is lower than the preset threshold, and the trained U-type neural network model is obtained.

[0069] In an exemplary embodiment of the present disclosure, by acquiring an image training set for a specified camera module, when training a Unet model, since the image training set is specially designed for application to a specified camera module, training the Unet model according to the targeted image training set can make the trained Unet model more targeted and directed, and when the trained Unet model is deployed to a digital image processing platform that applies a specified camera module, the image can be processed faster. Moreover, based on the de-mosaicing of the image in the specified camera module, the color image obtained is more realistic and the noise intensity is weakened, so that the color image output by the Unet model has higher clarity, and the image output can still have better color reproduction in extremely dark and extremely bright extreme lighting environments.

[0070] Based on the same inventive concept, the present disclosure also provides an image processing device.

[0071] It is understandable that, in order to realize the above functions, the application control device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0072] Figure 4 FIG. 4 is a block diagram of an image processing device according to an exemplary embodiment. Figure 4 , the image processing device includes an acquisition unit 401 and a processing unit 402.

[0073] Among them, the acquisition unit 401 is configured to acquire a single-channel grayscale image; the processing unit 402 is configured to interpolate the green channel pixels of the single-channel grayscale image through an interpolation algorithm to obtain the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation of the single-channel grayscale image, and use a pre-trained neural network model to adaptively interpolate and fuse the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation to obtain the green channel image of the single-channel grayscale image, and use the green channel image as a reference template to use the interpolation algorithm to complete the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain a color image.

[0074] In one example, the image processing device also includes: a determination unit 403, configured to determine an image training set, the image training set including a color training image; the processing unit 402 is also configured to: preprocess the color training image in the image training set to obtain a preprocessed color training image; determine the preprocessed color training image as a comparison image of the output image of the neural network model; mosaic process the comparison image to obtain a single-channel grayscale training image corresponding to the comparison image; and a training unit 404, configured to train the neural network model based on the single-channel grayscale training image and the comparison image.

[0075] In one example, the image training set includes one or more of: high-definition images acquired based on a public high-definition image set, high-frequency texture images, images taken by a designated camera module, and images taken by a single-lens reflex camera.

[0076] In one example, the processing unit 402 is further configured to: perform brightness and noise removal and edge enhancement processing on the comparison image to obtain a processed comparison image.

[0077] In one example, the pre-trained neural network model is a U-type neural network model.

[0078] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0079] Figure 5 1 is a block diagram of an apparatus 500 for image processing according to an exemplary embodiment. For example, the apparatus 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0080] Reference Figure 5 , the device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 505 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0081] The processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0082] The memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0083] The power supply component 505 provides power to the various components of the device 500. The power supply component 505 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 500.

[0084] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0085] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the device 500 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0086] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0087] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the device 500, the sensor assembly 514 can also detect the position change of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and the temperature change of the device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0088] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0089] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0090] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by the processor 520 of the device 500 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0091] It is further understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0092] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0093] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0094] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0095] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get a single-channel grayscale image; Interpolating the green channel pixels of the single-channel grayscale image by an interpolation algorithm to obtain horizontal pixel interpolation of the green channel and vertical pixel interpolation of the green channel of the single-channel grayscale image; Using a pre-trained neural network model, adaptively interpolating and fusing the horizontal pixel interpolation of the green channel and the vertical pixel interpolation of the green channel to obtain a green channel image of the single-channel grayscale image; Taking the green channel image as a reference template, the interpolation algorithm is used to complement the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain the red channel image and the blue channel image of the single-channel grayscale image, and a color image is obtained according to the green channel image, the red channel image and the blue channel image of the single-channel grayscale image.

2. The image processing method according to claim 1, characterized in that: The method further comprises: Determine an image training set, wherein the image training set includes color training images; Preprocessing the color training images in the image training set to obtain preprocessed color training images; Determine the preprocessed color training image as a comparison image of the output image of the neural network model; Performing mosaic processing on the comparison image to obtain a single-channel grayscale training image corresponding to the comparison image; The neural network model is trained based on the single-channel grayscale training image and the comparison image.

3. The image processing method according to claim 2, characterized in that: The image training set includes: One or more of high-definition images acquired based on a public high-definition image set, high-frequency texture images, images taken by a designated camera module, and images taken by a single-lens reflex camera.

4. The image processing method according to claim 2, characterized in that: The comparison image is obtained after brightness and noise are removed and edge enhancement is performed.

5. The image processing method according to any one of claims 1 to 4, characterized in that: The neural network model is a U-type neural network model.

6. An image processing device, characterized in that: The device comprises: An acquisition unit is configured to acquire a single-channel grayscale image; a processing unit configured to interpolate the green channel pixels of the single-channel grayscale image through an interpolation algorithm to obtain the green channel horizontal pixel interpolation and the green channel vertical pixel interpolation of the single-channel grayscale image, and Using a pre-trained neural network model, adaptively interpolating and fusing the horizontal pixel interpolation of the green channel and the vertical pixel interpolation of the green channel to obtain a green channel image of the single-channel grayscale image, and Taking the green channel image as a reference template, the interpolation algorithm is used to complement the red channel pixels and the blue channel pixels of the single-channel grayscale image to obtain the red channel image and the blue channel image of the single-channel grayscale image, and a color image is obtained according to the green channel image, the red channel image and the blue channel image of the single-channel grayscale image.

7. The image processing device according to claim 6, characterized in that: The device also includes: A determination unit, configured to determine an image training set, wherein the image training set includes color training images; The processing unit is further configured to: Preprocessing the color training images in the image training set to obtain preprocessed color training images; Determine the preprocessed color training image as a comparison image of the output image of the neural network model; Performing mosaic processing on the comparison image to obtain a single-channel grayscale training image corresponding to the comparison image; A training unit is configured to train the neural network model based on the single-channel grayscale training image and the comparison image.

8. The image processing device according to claim 7, characterized in that: The image training set includes: One or more of high-definition images acquired based on a public high-definition image set, high-frequency texture images, images taken by a designated camera module, and images taken by a single-lens reflex camera.

9. The image processing device according to claim 7, characterized in that: The comparison image is obtained after brightness and noise are removed and edge enhancement is performed.

10. The image processing device according to any one of claims 6 to 9, characterized in that: The pre-trained neural network model is a U-type neural network model.

11. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the image processing method according to any one of claims 1-5.

12. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 5 is executed.

Images