De-mosaicing method, device, electronic device and storage medium
By using neural networks in RGBIR image processing to eliminate interference components in RGBIR images, the problems of interpolation error and expert experience dependence in the prior art are solved, and high-quality RGB and IR image restoration is achieved, suitable for different models of RGBIR sensors .
Patent Information
- Application Number
- CN202110919874.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-08-11
AI Technical Summary
The existing solution mosaic algorithms are prone to introduce interpolation errors and rely on expert experience in RGBIR image processing, resulting in low quality of target RGB images and target IR images and cannot be applied to different models of RGBIR sensors.
The preprocessed RGBIR image is processed using the first neural network and the second neural network to eliminate interference components in the channel pixel points and predict pixel values, avoid interpolation errors, and automatically complete the demosaic process without manual correction.
Improves the quality of the target RGB image and the target IR image, and is suitable for any RGBIR image sensor, reducing costs and improving the signal-to-noise ratio and color reduction of the image.
Smart Images

Figure CN113781326B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a demosaicing method, device, electronic device and storage medium. Background Art
[0002] The RGBIR image sensor is a sensor that can sense both infrared information and visible light information. The RGBIR image sensor replaces the pixels covering the green filter in the Bayer array with pixels covering the near-infrared filter, so that information from both visible light and near-infrared light bands can be sensed simultaneously through one image sensor. The images required for computer vision tasks are target RGB images and / or target IR images. Therefore, it is necessary to obtain the target RGB image and / or target IR image based on the RGBIR image captured by the RGBIR image sensor. Obtaining the target RGB image and / or target IR image based on the RGBIR image captured by the RGBIR image sensor is called demosaicing.
[0003] At present, the commonly used demosaicing algorithm is: interpolating the RGBIR image collected by the RGBIR image sensor to generate an RGB image and an IR (Infrared Radiation) image. Performing color correction on the RGB image generated by interpolation and / or the IR image generated by interpolation, eliminating interference information in the RGB image generated by interpolation and the IR image generated by interpolation, completing demosaicing, and obtaining the target RGB image and the target IR image.
[0004] The commonly used de-mosaicing algorithm will introduce a large interpolation error, resulting in low quality of the target RGB image and / or target IR image. In addition, manual correction is highly dependent on expert experience, which inevitably leads to correction errors, resulting in low quality of the target RGB image and target IR image. The commonly used de-mosaicing algorithm is strongly related to the color filter array of the RGBIR sensor. The de-mosaicing algorithm applicable to the color filter array of one model of RGBIR sensor cannot be applied to the color filter array of another model of RGBIR sensor. A de-mosaicing algorithm must be developed separately for each model of RGBIR sensor, resulting in high costs. Summary of the invention
[0005] Embodiments of the present application provide a mosaic demodulation method, device, electronic device, and storage medium.
[0006] The present application provides a method for de-mosaicing, including:
[0007] Acquire a target RGBIR image; wherein the target RGBIR image is a preprocessed RGBIR image, and the preprocessing includes dark level compensation;
[0008] The target RGBIR image is first processed by using a first neural network to obtain a target IR image; and / or, the target RGBIR image is second processed by using a second neural network to obtain a target RGB image; wherein the first processing includes: eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image, and predicting the pixel value of the color channel pixel point in the IR channel, and the second processing includes: for each color channel, eliminating the near-infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and predicting the pixel value of the pixel point other than the color channel pixel point in the color channel.
[0009] The present application provides a de-mosaicing device, including:
[0010] An acquisition unit is configured to acquire a target RGBIR image; wherein the target RGBIR image is a preprocessed RGBIR image, and the preprocessing includes dark level compensation;
[0011] The demosaicing unit is configured to perform a first processing on the target RGBIR image using a first neural network to obtain a target IR image; and / or, perform a second processing on the target RGBIR image using a second neural network to obtain a target RGB image; wherein the first processing includes: eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image, and predicting the pixel value of the color channel pixel point in the IR channel, and the second processing includes: for each color channel, eliminating the near-infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and predicting the pixel value of the pixel point other than the color channel pixel point in the color channel.
[0012] The demosaicing method and device provided in the embodiment of the present application use the first neural network and / or the second neural network to complete the demosaicing of the target RGBIR image to obtain the target IR image and / or the target RGB image. In the demosaicing process, the component serving as interference information in the pixel value corresponding to the corresponding channel pixel point in the target RGBIR image is eliminated, the component serving as interference information is the visible light component or the near-infrared light component, and the pixel value of the corresponding pixel point in the corresponding channel is predicted. In the demosaicing process, the component pixel value with the corresponding interference information is not used for interpolation, and no interpolation error is generated. At the same time, the demosaicing process is automatically completed by the first neural network and / or the second neural network without relying on expert experience. The correction is performed manually, and no correction error is generated. Therefore, the adverse effects of interpolation error and correction error on demosaicing are avoided, and the image quality of the obtained target IR image and / or the image quality of the target RGB image is improved, which provides favorable conditions for the subsequent generation of IR images with high signal-to-noise ratio and good color reproduction and / or RGB images with good color reproduction. On the other hand, the first neural network and / or the second neural network can eliminate the component of interference information in the pixel value corresponding to the pixel point of any channel in the target RGBIR image, and predict the pixel value of any pixel point in the target RGBIR image in the corresponding channel. It is not affected by the color filter array of the sensor and is suitable for demosaicing RGBIR images collected by any RGBIR image sensor. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0014] Figure 1 A flowchart of a de-mosaicing method provided in an embodiment of the present application is shown;
[0015] Figure 2 A schematic diagram of a structure applicable to the first process or the second process is shown;
[0016] Figure 3 Another schematic diagram of a structure applicable to the first process or the second process is shown;
[0017] Figure 4 A structural block diagram of a demosaicing device provided in an embodiment of the present application is shown;
[0018] Figure 5 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0019] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0021] Figure 1 A flowchart of a mosaic removal method provided in an embodiment of the present application is shown, and the method includes:
[0022] Step 101, obtaining a target RGBIR image.
[0023] In the present application, the target RGBIR image is a pre-processed RGBIR image, and the pre-processing includes dark level compensation.
[0024] The preprocessed RGBIR image can be obtained by preprocessing the RGBIR image collected by the RGBIR sensor by performing dark level compensation in advance. Since the RGBIR sensor itself has a dark level, the RGBIR image collected by the RGBIR sensor is affected by the dark level of the RGBIR sensor. Therefore, it is necessary to eliminate the influence of the dark level of the RGBIR sensor on the RGBIR image collected by the RGBIR sensor. The preprocessing device can preprocess the RGBIR image collected by the RGBIR sensor to perform dark level compensation to obtain the preprocessed RGBIR image.
[0025] In some embodiments, acquiring a target RGBIR image includes: acquiring an original RGBIR image; and preprocessing the original RGBIR image to obtain a target RGBIR image. The original RGBIR image is an RGBIR image acquired by an RGBIR sensor. When preprocessing the original RGBIR image, dark level compensation may be performed on the original RGBIR image to obtain a dark level compensated RGBIR image, and the dark level compensated RGBIR image may be used as the target RGBIR image.
[0026] The dark level compensation of the original RGBIR image can be expressed as:
[0027]
[0028] Among them, BLCbayer represents the dark level of the RGBIR sensor, and the dark level of the RGBIR sensor can be provided by the RGBIR sensor manufacturer.
[0029] R, G, B, and IR on the right side of the equation represent the original pixel value of each pixel in the R channel, the original pixel value of the G channel, the original pixel value of the B channel, and the original pixel value of the IR channel, respectively, and R, G, B, and IR on the left side of the equation represent the pixel value of the R channel pixel after dark level compensation, the pixel value of the G channel pixel after dark level compensation, the pixel value of the B channel pixel after dark level compensation, and the pixel value of the IR channel pixel after dark level compensation, respectively.
[0030] In the present application, an R pixel corresponds to a red filter in the array of an RGBIR sensor. For an R pixel, the position of the R pixel in the RGBIR image is the same as the position of the red filter corresponding to the R pixel in the array of the RGBIR sensor. A G pixel corresponds to a green filter in the array of an RGBIR sensor. For a G pixel, the position of the G pixel in the RGBIR image is the same as the position of the green filter corresponding to the G pixel in the array of the RGBIR sensor. A B pixel corresponds to a blue filter in the array of an RGBIR sensor. For a B pixel, the position of the B pixel in the RGBIR image is the same as the position of the blue filter corresponding to the B pixel in the array of the RGBIR sensor. An IR pixel corresponds to a near-infrared filter in the array of an RGBIR sensor. For an IR pixel, the position of the IR pixel in the RGBIR image is the same as the position of the near-infrared filter corresponding to the IR pixel in the array of the RGBIR sensor.
[0031] In a specific implementation, the preprocessing may further include: image normalization. Image normalization may make the pixel values in the corresponding image obtained by image normalization conform to a certain distribution, so as to facilitate the neural network used to process the corresponding image to process the corresponding image obtained by image normalization. The RGBIR image after dark level compensation may be subjected to image normalization to obtain a target RGBIR image.
[0032] Step 102: Perform a first process on the target RGBIR image using a first neural network to obtain a target IR image, and / or perform a second process on the target RGBIR image using a second neural network to obtain a target RGB image.
[0033] In the present application, the first neural network can be a convolutional neural network, and the first processing includes: eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image, and predicting the pixel value of the color channel pixel point in the IR channel.
[0034] Due to the crosstalk between visible light and near-infrared light, the pixel value of the IR channel pixel in the RGBIR image is composed of the part affected by near-infrared light, namely the near-infrared light component, and the part affected by visible light, namely the visible light component. By eliminating the visible light component in the pixel value corresponding to the IR channel pixel, the near-infrared light component in the pixel value corresponding to the IR channel pixel can be obtained.
[0035] In the present application, the pixel value of each pixel in the target IR image does not have a visible light component.
[0036] For each IR channel pixel in the target RGBIR image, the target IR image includes a pixel corresponding to the IR channel pixel, the pixel value corresponding to the IR channel pixel is the near-infrared light component in the pixel value corresponding to the IR channel pixel, and the position of the pixel corresponding to the IR channel pixel in the target IR image is the same as the position of the IR channel pixel in the target RGBIR image.
[0037] For each R channel pixel in the target RGBIR image, the target IR image includes a pixel corresponding to the R channel pixel, the pixel value corresponding to the R channel pixel is the predicted pixel value of the R channel pixel in the IR channel, and the position of the pixel corresponding to the R channel pixel in the target IR image is the same as the position of the R channel pixel in the target RGBIR image.
[0038] For each G channel pixel in the target RGBIR image, the target IR image includes a pixel corresponding to the G channel pixel, the pixel value corresponding to the G channel pixel is the predicted pixel value of the G channel pixel in the IR channel, and the position of the pixel corresponding to the G channel pixel in the target IR image is the same as the position of the G channel pixel in the target RGBIR image.
[0039] For each B channel pixel in the target RGBIR image, the target IR image includes a pixel corresponding to the B channel pixel, the pixel value corresponding to the B channel pixel is the predicted pixel value of the B channel pixel in the IR channel, and the position of the pixel corresponding to the B channel pixel in the target IR image is the same as the position of the B channel pixel in the target RGBIR image.
[0040] In the present application, before acquiring the target IR image using the first neural network, the first neural network is trained in advance using a plurality of image pairs for training the first neural network.
[0041] The image pair used to train the first neural network includes an RGBIR image and an IR image without visible light interference. The IR image without visible light interference can be collected by a sensor for collecting IR images without visible light interference, for example, the IR image without visible light interference can be collected by an IR sensor. For each image pair used to train the first neural network, the RGBIR image and the IR image without visible light interference in the image pair can be obtained by simultaneously photographing the same object at the same time by the RGBIR image sensor and the sensor for collecting IR images without visible light interference. At the same time, the position of the RGBIR image sensor is the same as the position of the sensor for collecting IR images without visible light interference. At the same time, the RGBIR image sensor photographs the same object to obtain the RGBIR image in the image pair, and at the same time, the IR image sensor for collecting IR images without visible light interference photographs the same object to obtain the IR image without visible light interference in the image pair.
[0042] Each time, the first neural network is trained using an image pair for training the first neural network. Each time, the image pair used for training the first neural network is different. Each time the first neural network is trained, the RGBIR image in the image pair used for training the first neural network is used as the input of the first neural network, and the IR image without visible light interference in the image pair used for training the first neural network is used as the label, i.e., Ground-Truth. The first neural network learns the association between the RGBIR image and the IR image without visible light interference. Specifically, the first neural network can learn the association between the pixel values of the pixel points at the same position in the RGBIR image and the IR image without visible light interference.
[0043] In the present application, the second neural network can be a convolutional neural network, and the second processing includes: for each color channel, eliminating the near-infrared light component in the pixel value corresponding to the pixel point of the color channel in the target RGBIR image, and predicting the pixel value of the pixel point in the color channel other than the pixel point of the color channel.
[0044] The R channel, G channel, and B channel are all color channels. Due to the crosstalk phenomenon between visible light and near-infrared light, the pixel values of the R channel pixels, G channel pixels, and B channel pixels in the RGBIR image are composed of the part affected by visible light, i.e., the visible light component, and the part affected by near-infrared light, i.e., the near-infrared light component.
[0045] By eliminating the near-infrared light component in the pixel value corresponding to the R channel pixel point, the visible light component in the pixel value corresponding to the R channel pixel point can be obtained. By eliminating the near-infrared light component in the pixel value corresponding to the G channel pixel point, the visible light component in the pixel value corresponding to the G channel pixel point can be obtained. By eliminating the near-infrared light component in the pixel value corresponding to the B channel pixel point, the visible light component in the pixel value corresponding to the B channel pixel point can be obtained.
[0046] In the present application, through the second processing, a target R channel image, a target G channel image, and a target B channel image can be obtained, and the target R channel image, the target G channel image, and the target B channel image constitute a target RGB image.
[0047] For each R channel pixel in the target RGBIR image, the target R channel image includes a pixel corresponding to the R channel pixel, the pixel value of the pixel corresponding to the R channel pixel is the visible light component in the pixel value corresponding to the R channel pixel, and the position of the pixel corresponding to the R channel pixel in the target R channel image is the same as the position of the R channel pixel in the target RGBIR image.
[0048] For each G channel pixel in the target RGBIR image, the target R channel image includes a pixel corresponding to the G channel pixel, the pixel value of the pixel corresponding to the G channel pixel is the predicted pixel value of the G channel pixel in the R channel, and the position of the pixel corresponding to the G channel pixel in the target R channel image is the same as the position of the G channel pixel in the target RGBIR image.
[0049] For each B channel pixel in the target RGBIR image, the target R channel image includes a pixel corresponding to the B channel pixel, the pixel value of the pixel corresponding to the B channel pixel is the predicted pixel value of the B channel pixel in the R channel, and the position of the pixel corresponding to the B channel pixel in the target R channel image is the same as the position of the B channel pixel in the target RGBIR image.
[0050] For each IR channel pixel in the target RGBIR image, the target R channel image includes a pixel corresponding to the IR channel pixel, the pixel value of the pixel corresponding to the IR channel pixel is the predicted pixel value of the IR channel pixel in the R channel, and the position of the pixel corresponding to the IR channel pixel in the target R channel image is the same as the position of the IR channel pixel in the target RGBIR image.
[0051] For each G channel pixel in the target RGBIR image, the target G channel image includes a pixel corresponding to the G channel pixel, the pixel value of the pixel corresponding to the G channel pixel is the visible light component in the pixel value corresponding to the G channel pixel, and the position of the pixel corresponding to the G channel pixel in the target G channel image is the same as the position of the G channel pixel in the target RGBIR image.
[0052] For each R channel pixel in the target RGBIR image, the target G channel image includes a pixel corresponding to the R channel pixel, the pixel value of the pixel corresponding to the R channel pixel is the predicted pixel value of the R channel pixel in the G channel, and the position of the pixel corresponding to the R channel pixel in the target G channel image is the same as the position of the R channel pixel in the target RGBIR image.
[0053] For each B channel pixel in the target RGBIR image, the target G channel image includes a pixel corresponding to the B channel pixel, the pixel value of the pixel corresponding to the B channel pixel is the predicted pixel value of the B channel pixel in the G channel, and the position of the pixel corresponding to the B channel pixel in the target G channel image is the same as the position of the B channel pixel in the target RGBIR image.
[0054] For each IR channel pixel in the target RGBIR image, the target G channel image includes a pixel corresponding to the IR channel pixel, the pixel value of the pixel corresponding to the IR channel pixel is the predicted pixel value of the IR channel pixel in the G channel, and the position of the pixel corresponding to the IR channel pixel in the target G channel image is the same as the position of the IR channel pixel in the target RGBIR image.
[0055] For each B-channel pixel in the target RGBIR image, the target B-channel image includes a pixel corresponding to the B-channel pixel, the pixel value of the pixel corresponding to the B-channel pixel is the visible light component in the pixel value corresponding to the B-channel pixel, and the position of the pixel corresponding to the B-channel pixel in the target B-channel image is the same as the position of the B-channel pixel in the target RGBIR image.
[0056] For each R channel pixel in the target RGBIR image, the target B channel image includes a pixel corresponding to the R channel pixel, the pixel value of the pixel corresponding to the R channel pixel is the predicted pixel value of the R channel pixel in the B channel, and the position of the pixel corresponding to the R channel pixel in the target B channel image is the same as the position of the R channel pixel in the target RGBIR image.
[0057] For each G channel pixel in the target RGBIR image, the target B channel image includes a pixel corresponding to the G channel pixel, the pixel value of the pixel corresponding to the G channel pixel is the predicted pixel value of the G channel pixel in the B channel, and the position of the pixel corresponding to the G channel pixel in the target B channel image is the same as the position of the R channel pixel in the target RGBIR image.
[0058] For each IR channel pixel in the target RGBIR image, the target B channel image includes a pixel corresponding to the IR channel pixel, the pixel value of the pixel corresponding to the IR channel pixel is the predicted pixel value of the IR channel pixel in the B channel, and the position of the pixel corresponding to the IR channel pixel in the target B channel image is the same as the position of the IR channel pixel in the target RGBIR image.
[0059] In the present application, before using the second neural network to acquire the target RGB image, the second neural network is trained in advance using a plurality of image pairs for training the second neural network. The image pairs for training the second neural network include an RGBIR image and an RGB image without near-infrared light interference. The RGB image without near-infrared light interference is acquired by a sensor for acquiring RGB images without near-infrared light interference. For example, the RGB image without near-infrared light interference can be acquired by an RGB sensor.
[0060] For each image pair used to train the second neural network, the RGBIR image and the RGB image without near-infrared interference in the image pair can be obtained by simultaneously photographing the same object at the same time by the RGBIR image sensor and the sensor for collecting the RGB image without near-infrared interference. At the same time, the position of the RGBIR image sensor is the same as the position of the sensor for collecting the RGB image without near-infrared interference. At the same time, the RGBIR image sensor photographs the same object to obtain the RGBIR image in the image pair, and at the same time, the sensor for collecting the RGB image without near-infrared interference photographs the same object to obtain the RGB image without near-infrared interference in the image pair.
[0061] Each time, the second neural network is trained using an image pair for training the second neural network. Each time, the image pair used for training the second neural network is different. Each time the second neural network is trained, the RGBIR image in the image pair for training the second neural network is used as the input of the second neural network, and the RGB image without near-infrared light interference in the image pair for training the second neural network is used as a label. During the training process, the second neural network learns the association relationship between the RGBIR image and the RGB image without near-infrared light interference. Specifically, the second neural network can learn the association relationship between the pixel values of the same position in the RGBIR image and the RGB image without near-infrared light interference.
[0062] In the present application, the first neural network and / or the second neural network are used to complete demosaicing to obtain a target IR image and / or a target RGB image. During the demosaicing process, the components of the pixel values corresponding to the corresponding channel pixel points in the target RGBIR image that serve as interference information are eliminated, and the components of the interference information are visible light components or near-infrared light components. The pixel values of the corresponding pixel points in the corresponding channels are predicted. During the demosaicing process, the component pixel values with the corresponding interference information are not used for interpolation, and no interpolation error will be generated. At the same time, the demosaicing process is automatically completed by the first neural network and / or the second neural network without relying on expert experience. Correction is performed manually, and no correction error will be generated. Therefore, the adverse effects of interpolation error and correction error on demosaicing are avoided, and the image quality of the obtained target IR image and / or the image quality of the target RGB image are improved, which provides favorable conditions for the subsequent generation of IR images with high signal-to-noise ratio and good color reproduction and / or RGB images with good color reproduction. On the other hand, the first neural network and / or the second neural network can eliminate the component of interference information in the pixel value corresponding to the pixel point of any channel in the target RGBIR image, and predict the pixel value of any pixel point in the target RGBIR image in the corresponding channel. It is not affected by the color filter array of the sensor and is suitable for demosaicing RGBIR images collected by any RGBIR image sensor.
[0063] In some embodiments, the size of the target IR image is consistent with the size of the target RGBIR image, and the size of the target RGB image is consistent with the size of the target RGBIR image.
[0064] In some embodiments, eliminating the visible light component of the pixel value of the IR channel pixel in the target RGBIR image, and predicting the pixel value of the color channel pixel in the IR channel include: performing feature extraction processing on the target RGBIR image to obtain a first feature, the first feature including: near-infrared light band information of the target RGBIR image; performing IR image reconstruction processing, the IR image reconstruction processing including: based on the first feature, eliminating the visible light component in the pixel value corresponding to the IR channel pixel to obtain the near-infrared light component in the pixel value corresponding to the IR channel pixel; based on the near-infrared light component in the pixel value corresponding to the IR channel pixel, predicting the pixel value of the color channel pixel in the IR channel.
[0065] In the present application, the first neural network may include a feature extraction module and a reconstruction module. The feature extraction module in the first neural network may perform feature extraction processing on the target RGBIR image to obtain the first feature. The reconstruction module in the first neural network may perform IR image reconstruction processing. The feature extraction module in the first neural network may include blocks and pooling layers, and the blocks may include multiple convolutional layers. The reconstruction module in the first neural network may include blocks and upsampling layers. The input of the feature extraction module in the first neural network is the target RGBIR image, the output of the feature extraction module in the first neural network is the first feature, the input of the reconstruction module in the first neural network is the first feature, and the output of the reconstruction module in the first neural network is the target IR image.
[0066] In some embodiments, for each color channel, eliminating the near-infrared light component in the pixel value corresponding to the pixel point of the color channel in the target RGBIR image, and predicting the pixel value of the pixel point in the color channel other than the pixel point of the color channel includes: performing feature extraction processing on the target RGBIR image to obtain a second feature, and the second feature includes: visible light band information of the target RGBIR image; performing RGB image reconstruction processing, and the RGB image reconstruction processing includes: for each color channel, based on the second feature, eliminating the near-infrared light component in the pixel value corresponding to the pixel point of the color channel in the target RGBIR image, and obtaining the visible light component in the pixel value corresponding to the pixel point of the color channel; based on the visible light component in the pixel value corresponding to the pixel point of the color channel, predicting the pixel value of the pixel point in the color channel other than the pixel point of the color channel.
[0067] In the present application, the first neural network may include a feature extraction module and a reconstruction module. The feature extraction module in the second neural network may perform feature extraction processing on the target RGBIR image to obtain the second feature. The reconstruction module in the second neural network may perform RGB image reconstruction processing. The feature extraction module in the second neural network may include blocks and pooling layers, and the blocks may include multiple convolutional layers. The reconstruction module in the second neural network may include blocks and upsampling layers. The input of the feature extraction module in the second neural network is the target RGBIR image, the output of the feature extraction module in the second neural network is the second feature, the input of the reconstruction module in the second neural network is the second feature, and the output of the reconstruction module in the second neural network is the target RGB image.
[0068] Please refer to Figure 2 , which shows a structural schematic diagram suitable for the first process or the second process.
[0069] like Figure 2 The structure in is the structure used for the first processing, Figure 2 The feature extraction module in refers to the feature extraction module in the first neural network. Figure 2 The reconstruction module in refers to the reconstruction module in the first neural network. Figure 2 The structure in is the structure used for the second processing, Figure 2 The feature extraction module in refers to the feature extraction module in the second neural network. Figure 2 The reconstruction module in refers to the reconstruction module in the second neural network. The feature extraction module includes block 1, block 2, and pooling layer. The reconstruction module includes: block 3, block 4, and upsampling layer. The input of the feature extraction module, that is, the input of block 1, is an RGBIR image. If Figure 2 The structure in is the structure for the first processing, and the output of the reconstruction module, that is, the output of block 4, is the target IR image. Figure 2 The structure in is the structure used for the second processing, and the output of the reconstruction module is the target RGB image.
[0070] Please refer to Figure 3 , which shows another structural schematic diagram suitable for the first process or the second process.
[0071] like Figure 3 The structure in is the structure used for the first processing, Figure 3 The feature extraction module in refers to the feature extraction module in the first neural network. Figure 3 The reconstruction module in refers to the reconstruction module in the first neural network. Figure 2 The structure in is the structure used for the second processing, Figure 3 The feature extraction module in refers to the feature extraction module in the second neural network. Figure 3The reconstruction module in refers to the reconstruction module in the second neural network.
[0072] The feature extraction module includes block 1, block 2, block 3, and multiple pooling layers. The reconstruction module includes block 4, block 5, block 6, and multiple upsampling layers. The input of the feature extraction module, i.e., the input of block 1, is an RGBIR image. Figure 3 The structure in is the structure for the first processing, and the output of the reconstruction module, that is, the output of the upsampling layer connected to block 6, is the target IR image. Figure 3 The structure in is the structure used for the second processing, and the output of the reconstruction module is the target RGB image.
[0073] Figure 3 The residual connection in is an optional connection. The residual connection is used to add the outputs of two layers to obtain a connection result, which is used as the input of a certain block. If the output of the pooling layer connected to block 1 and the output of the upsampling layer connected to block 5 are added through a residual connection, the connection result is used as the input of block 6. If the output of the pooling layer connected to block 2 and the output of the upsampling layer connected to block 4 are added through a residual connection, the connection result is used as the input of block 5.
[0074] In some embodiments, before acquiring the target RGBIR image, it also includes: acquiring multiple first training image pairs, the first training image pairs including: a first RGBIR training image and a label image corresponding to the first RGBIR training image, the label image corresponding to the first RGBIR training image is an IR image without visible light interference, and the label image corresponding to the first RGBIR training image is obtained by performing a first integral operation on the multispectral image corresponding to the first RGBIR training image; and using the multiple first training image pairs to train the first neural network.
[0075] In the present application, the image used to train the first neural network may be referred to as a first RGBIR training image.
[0076] Each time the first neural network is trained, a first training image pair is used to train the first neural network.
[0077] The first training image pair includes: a first RGBIR training image and a label image corresponding to the first RGBIR training image. The first RGBIR training image in the first training image pair used for training the first neural network is different each time.
[0078] The label image corresponding to the first RGBIR training image is an IR image without visible light interference, that is, in the label image corresponding to the first RGBIR training image, the pixel value of each pixel does not have a visible light component.
[0079] For each first RGBIR training image, the multispectral image corresponding to the first RGBIR training image can describe the IR response value of each pixel point in the label image corresponding to the first RGBIR training image in multiple bands, and a first integral operation can be performed on the multispectral image corresponding to the first RGBIR training image. Through the first integral operation, for each pixel point in the label image corresponding to the first RGBIR training image, according to the IR response value of the pixel point in multiple bands, the pixel value of the pixel point can be determined. After determining the pixel value of each pixel point in the label image corresponding to the first RGBIR training image, the label image corresponding to the first RGBIR training image can be obtained.
[0080] For each first RGBIR training image, a multispectral image corresponding to the first RGBIR training image can be collected while the first RGBIR training image is collected in advance. The first RGBIR training image and the multispectral image corresponding to the first RGBIR training image can be obtained by simultaneously photographing the same object by an RGBIR image sensor and a multispectral image sensor at the same time. At the same time, the position of the RGBIR image sensor is the same as the position of the multispectral image sensor.
[0081] In a training process, a first RGBIR training image in a first training image pair is input into the first neural network to obtain a predicted IR image output by the first neural network. A square loss function is used to calculate the loss between the predicted IR image and the label image corresponding to the first RGBIR training image. Based on the loss between the predicted IR image and the label image corresponding to the first RGBIR training image, the parameter values of the parameters of the infrared light band information extraction network are updated.
[0082] The multispectral image corresponding to the first RGBIR training image can describe the IR response values of each pixel in the label image corresponding to the first RGBIR training image in multiple bands. By performing a first integration operation on the multispectral image corresponding to the first RGBIR training image, the pixel value of each pixel in the label image corresponding to the first RGBIR training image can be accurately determined, and the obtained label image corresponding to the first RGBIR training image has high accuracy.
[0083] In some embodiments, acquiring multiple first training image pairs includes: acquiring first RGBIR training images of multiple different scenes and multispectral images corresponding to the first RGBIR training images of multiple different scenes; for each acquired first RGBIR training image, performing a first integral operation on the multispectral image corresponding to the first RGBIR training image to obtain a label image corresponding to the first RGBIR training image; and determining the first RGBIR training image and the label image corresponding to the first RGBIR training image as a first training image pair.
[0084] The multiple different scenes may refer to multiple scenes with different lighting conditions. For each scene, the first RGBIR training image of the scene refers to the first RGBIR training image collected in advance in the scene. For each scene, multiple first RGBIR training images of the scene may be collected in advance in the scene, and for each first RGBIR training image of the scene, a multispectral image corresponding to the first RGBIR training image may be collected at the same time as the first RGBIR training image is collected.
[0085] For each first RGBIR training image obtained, a first integration operation is performed on the multispectral image corresponding to the first RGBIR training image to obtain a label image corresponding to the first RGBIR training image, and the first RGBIR training image and the label image corresponding to the first RGBIR training image are combined into a first training image pair.
[0086] In the present application, multiple first training image pairs may include first RGBIR training images of multiple different scenes and label images corresponding to the first RGBIR training images of multiple different scenes. The first neural network may be trained using the first RGBIR training images of multiple different scenes and label images corresponding to the first RGBIR training images of multiple different scenes. After completing the training of the first neural network, the first neural network is suitable for processing RGBIR images collected in any one of the multiple scenes.
[0087] In some embodiments, before acquiring the target RGBIR image, it also includes: acquiring multiple second training image pairs, the second training image pairs including: a second RGBIR training image, and a label image corresponding to the second RGBIR training image, wherein the label image corresponding to the second RGBIR training image is an RGB image without near-infrared light interference, and the label image corresponding to the second RGBIR training image is obtained by performing a second integral operation on the multispectral image corresponding to the second RGBIR training image; and using the multiple second training image pairs to train the second neural network.
[0088] In the present application, the image used to train the second neural network may be referred to as a second RGBIR training image.
[0089] Each time the second neural network is trained, a second training image pair is used to train the second neural network. The second training image pair includes: a second RGBIR training image and a label image corresponding to the second RGBIR training image. Each time the second neural network is trained, the second RGBIR training image in the second training image pair is different.
[0090] The label image corresponding to the second RGBIR training image is an RGB image without near-infrared light interference, that is, in the label image corresponding to the second RGBIR training image, the pixel value of each pixel does not have an IR component.
[0091] For each second RGBIR training image, the multispectral image corresponding to the second RGBIR training image can describe the visible light response value of each pixel point in the label image corresponding to the second RGBIR training image in multiple bands, and the second integral operation can be performed on the multispectral image corresponding to the second RGBIR training image. Through the second integral operation, the pixel value of each pixel point in the label image corresponding to the second RGBIR training image can be determined according to the visible light response value of the pixel point in multiple bands. After determining the pixel value of each pixel point in the label image corresponding to the second RGBIR training image, the label image corresponding to the second RGBIR training image can be obtained.
[0092] For each second RGBIR training image, a multispectral image corresponding to the second RGBIR training image can be collected while the second RGBIR training image is collected in advance. The second RGBIR training image and the multispectral image corresponding to the second RGBIR training image can be obtained by simultaneously photographing the same object by an RGBIR image sensor and a multispectral image sensor at the same time. At the same time, the position of the RGBIR image sensor is the same as the position of the multispectral image sensor.
[0093] In a training process, a second RGBIR training image in a second training image pair is input into the second neural network to obtain a predicted RGB image output by the second neural network. The loss between the predicted RGB image and the label image corresponding to the second RGBIR training image is calculated using a square loss function. Based on the loss between the predicted RGB image and the label image corresponding to the second RGBIR training image, back propagation is performed to update the parameter values of the parameters of the second neural network.
[0094] The multispectral image corresponding to the second RGBIR training image can describe the visible light response value of each pixel point in the label image corresponding to the second RGBIR training image in multiple bands. By performing a second integration operation on the multispectral image corresponding to the second RGBIR training image, the pixel value of each pixel point in the label image corresponding to the second RGBIR training image can be accurately determined, and the obtained label image corresponding to the second RGBIR training image has high accuracy.
[0095] In some embodiments, acquiring multiple second training image pairs includes: acquiring second RGBIR training images of multiple different scenes and multispectral images corresponding to multiple different second RGBIR training images; for each acquired second RGBIR training image, performing a second integral operation on the multispectral image corresponding to the second RGBIR training image to obtain a label image corresponding to the second RGBIR training image; and combining the second RGBIR training image and the label image corresponding to the second RGBIR training image into a second training image pair.
[0096] The multiple scenes may refer to multiple scenes with different lighting conditions. For each scene, the second RGBIR training image of the scene refers to a second RGBIR training image collected in advance in the scene. For each scene, multiple second RGBIR training images may be collected in advance in the scene, and for each second RGBIR training image of the scene, a multispectral image corresponding to the second RGBIR training image may be collected while collecting the second RGBIR training image.
[0097] For each acquired second RGBIR training image, a second integration operation is performed on the multispectral image corresponding to the second RGBIR training image to obtain a label image corresponding to the second RGBIR training image, and the second RGBIR training image and the label image corresponding to the second RGBIR training image are combined into a second training image pair.
[0098] In the present application, the multiple second training image pairs may include second RGBIR training images of multiple different scenes and label images corresponding to the second RGBIR training images of multiple different scenes. The second neural network may be trained using the second RGBIR training images of multiple different scenes and label images corresponding to the second RGBIR training images of multiple different scenes. After completing the training of the second neural network, the second neural network is suitable for processing RGBIR images collected in any one of the multiple scenes.
[0099] Please refer to Figure 4, which shows a structural block diagram of a de-mosaicing device provided in an embodiment of the present application. The de-mosaicing device comprises: an acquisition unit 401 and a de-mosaicing unit 402.
[0100] The acquisition unit 401 is configured to acquire a target RGBIR image; wherein the target RGBIR image is a pre-processed RGBIR image, and the pre-processing includes dark level compensation;
[0101] The demosaicing unit 402 is configured to perform a first processing on the target RGBIR image using a first neural network to obtain a target IR image; and / or, perform a second processing on the target RGBIR image using a second neural network to obtain a target RGB image; wherein the first processing includes: eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image, and predicting the pixel value of the color channel pixel point in the IR channel, and the second processing includes: for each color channel, eliminating the near-infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and predicting the pixel value of the pixel point other than the color channel pixel point in the color channel.
[0102] In some embodiments, the demosaicing unit 402 is further configured to perform feature extraction processing on the target RGBIR image to obtain a first feature, wherein the first feature includes: near-infrared light band information of the target RGBIR image; perform IR image reconstruction processing, wherein the IR image reconstruction processing includes: based on the first feature, eliminating the visible light component in the pixel value corresponding to the IR channel pixel point to obtain the near-infrared light component in the pixel value corresponding to the IR channel pixel point; based on the near-infrared light component in the pixel value corresponding to the IR channel pixel point, predicting the pixel value of the color channel pixel point in the IR channel.
[0103] In some embodiments, the demosaicing unit 202 is further configured to perform feature extraction processing on the target RGBIR image to obtain a second feature, wherein the second feature includes: visible light band information of the target RGBIR image; perform RGB image reconstruction processing, wherein the RGB image reconstruction processing includes: for each color channel, based on the second feature, eliminating the near-infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image to obtain the visible light component in the pixel value corresponding to the color channel pixel point; based on the visible light component in the pixel value corresponding to the color channel pixel point, predicting the pixel value of the pixel point other than the color channel pixel point in the color channel.
[0104] In some embodiments, the size of the target IR image is consistent with the size of the target RGBIR image, and the size of the target RGB image is consistent with the size of the target RGBIR image.
[0105] In some embodiments, the demosaicing device comprises:
[0106] The first training unit is configured to obtain a plurality of first training image pairs before obtaining a target RGBIR image, the first training image pairs comprising: a first RGBIR training image and a label image corresponding to the first RGBIR training image, the label image being an IR image without visible light interference, the label image being obtained by performing a first integral operation on a multispectral image corresponding to the first RGBIR training image; and train a first neural network using the plurality of first training image pairs.
[0107] In some embodiments, the first training unit is further configured to acquire first RGBIR training images of multiple different scenes and multispectral images corresponding to the first RGBIR training images of multiple different scenes; for each acquired first RGBIR training image, perform a first integral operation on the multispectral image corresponding to the first RGBIR training image to obtain a label image corresponding to the first RGBIR training image; and determine the first RGBIR training image and the label image corresponding to the first RGBIR training image as a first training image pair.
[0108] In some embodiments, in some embodiments, the demosaicing device includes:
[0109] The second training unit is configured to obtain a plurality of second training image pairs before obtaining the target RGBIR image, the second training image pairs comprising: a second RGBIR training image and a label image corresponding to the second RGBIR training image, wherein the label image corresponding to the second RGBIR training image is an RGB image without near-infrared light interference, and the label image corresponding to the second RGBIR training image is obtained by performing a second integral operation on the multispectral image corresponding to the second RGBIR training image; and train the second neural network using the plurality of second training image pairs.
[0110] In some embodiments, the second training unit is further configured to acquire second RGBIR training images of multiple different scenes and multispectral images corresponding to the second RGBIR training images of multiple different scenes; for each acquired second RGBIR training image, perform a second integral operation on the multispectral image corresponding to the second RGBIR training image to obtain a label image corresponding to the second RGBIR training image; and combine the second RGBIR training image and the label image corresponding to the second RGBIR training image into a second training image pair.
[0111] In some embodiments, the demosaicing device further comprises:
[0112] The preprocessing unit is configured to obtain an original RGBIR image; and perform the preprocessing on the original RGBIR image to obtain the target RGBIR image.
[0113] Any step and any specific operation in any step in the embodiment of the demosaicing method provided in the present application can be completed by the corresponding unit in the demosaicing device. The process of the corresponding operation completed by each unit in the demosaicing device refers to the process of the corresponding operation described in the embodiment of the demosaicing method.
[0114] By completing demosaicing through a demosaicing device, a target IR image and / or a target RGB image can be obtained. During the demosaicing process, components serving as interference information in pixel values corresponding to corresponding channel pixel points in the target RGBIR image are eliminated, where the components serving as interference information are visible light components or near-infrared light components, and pixel values of corresponding pixel points in corresponding channels are predicted. During the demosaicing process, component pixel values having corresponding interference information are not used for interpolation, and interpolation errors will not be generated. Meanwhile, the demosaicing process is automatically completed by the first neural network and / or the second neural network without relying on expert experience, and correction is performed manually, and correction errors will not be generated. Thus, the adverse effects of interpolation errors and correction errors on demosaicing are avoided, and the image quality of the obtained target IR image and / or the image quality of the target RGB image is improved, providing favorable conditions for the subsequent generation of IR images with high signal-to-noise ratio and good color reproduction and / or RGB images with good color reproduction. On the other hand, the first neural network and / or the second neural network can eliminate the component of interference information in the pixel value corresponding to the pixel point of any channel in the target RGBIR image, and predict the pixel value of any pixel point in the target RGBIR image in the corresponding channel. It is not affected by the color filter array of the sensor and is suitable for demosaicing RGBIR images collected by any RGBIR image sensor.
[0115] Figure 5 5 is a block diagram of a structure of an electronic device provided by this embodiment. The electronic device includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532, which is used to store instructions that can be executed by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform the above method.
[0116] The electronic device may also include a power supply component 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input / output (I / O) interface 558. The electronic device may operate based on an operating system stored in the memory 532, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.
[0117] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0118] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0119] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for de-mosaicing, characterized in that: The method comprises: Acquire a target RGBIR image; wherein the target RGBIR image is a preprocessed RGBIR image, and the preprocessing includes dark level compensation; Performing a first processing on the target RGBIR image using a first neural network to obtain a target IR image; and / or performing a second processing on the target RGBIR image using a second neural network to obtain a target RGB image; The first processing includes: eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image, and predicting the pixel value of the color channel pixel point in the IR channel, and the second processing includes: for each color channel, eliminating the near infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and predicting the pixel value of the pixel point other than the color channel pixel point in the color channel; Before acquiring the target RGBIR image, the method further includes: Acquire a plurality of first training image pairs, the first training image pairs comprising: a first RGBIR training image and a label image corresponding to the first RGBIR training image, the label image being an IR image without visible light interference, the label image being obtained by performing a first integral operation on a multispectral image corresponding to the first RGBIR training image; training a first neural network using a plurality of first training image pairs; Before acquiring the target RGBIR image, the method further includes: Acquire a plurality of second training image pairs, the second training image pairs comprising: a second RGBIR training image and a label image corresponding to the second RGBIR training image, wherein the label image corresponding to the second RGBIR training image is an RGB image without near-infrared light interference, and the label image corresponding to the second RGBIR training image is obtained by performing a second integral operation on the multispectral image corresponding to the second RGBIR training image; A second neural network is trained using a plurality of second training image pairs.
2. The method according to claim 1, characterized in that The eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image and predicting the pixel value of the color channel pixel point in the IR channel includes: Performing feature extraction processing on the target RGBIR image to obtain a first feature, wherein the first feature includes: near infrared light band information of the target RGBIR image; Perform IR image reconstruction processing, the IR image reconstruction processing comprising: based on the first feature, eliminating the visible light component in the pixel value corresponding to the IR channel pixel point, and obtaining the near-infrared light component in the pixel value corresponding to the IR channel pixel point; based on the near-infrared light component in the pixel value corresponding to the IR channel pixel point, predicting the pixel value of the color channel pixel point in the IR channel.
3. The method according to claim 1, characterized in that: For each color channel, eliminating the near infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and predicting the pixel value of the pixel point in the color channel other than the color channel pixel point includes: Performing feature extraction processing on the target RGBIR image to obtain a second feature, wherein the second feature includes: visible light band information of the target RGBIR image; Perform RGB image reconstruction processing, the RGB image reconstruction processing comprising: for each color channel, based on the second feature, eliminating the near-infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and obtaining the visible light component in the pixel value corresponding to the color channel pixel point; based on the visible light component in the pixel value corresponding to the color channel pixel point, predicting the pixel value of the pixel point in the color channel other than the color channel pixel point.
4. The method according to any one of claims 1 to 3, characterized in that: The size of the target IR image is consistent with the size of the target RGBIR image, and the size of the target RGB image is consistent with the size of the target RGBIR image.
5. The method according to claim 1, characterized in that Acquiring a plurality of first training image pairs comprises: Acquire first RGBIR training images of multiple different scenes and multispectral images corresponding to the first RGBIR training images of multiple different scenes; For each first RGBIR training image obtained, performing a first integration operation on a multispectral image corresponding to the first RGBIR training image to obtain a label image corresponding to the first RGBIR training image; The first RGBIR training image and a label image corresponding to the first RGBIR training image are determined as a first training image pair.
6. The method according to claim 1, characterized in that Acquiring a plurality of second training image pairs comprises: Acquire a plurality of second RGBIR training images of different scenes and multispectral images corresponding to the second RGBIR training images of different scenes; For each acquired second RGBIR training image, a second integral operation is performed on the multispectral image corresponding to the second RGBIR training image to obtain a label image corresponding to the second RGBIR training image; and the second RGBIR training image and the label image corresponding to the second RGBIR training image are combined into a second training image pair.
7. The method according to claim 1, characterized in that Obtaining the target RGBIR image includes: Get the original RGBIR image; The original RGBIR image is preprocessed to obtain the target RGBIR image.
8. A de-mosaicing device, characterized in that: The device comprises: An acquisition unit is configured to acquire a target RGBIR image; wherein the target RGBIR image is a preprocessed RGBIR image, and the preprocessing includes dark level compensation; a demosaicing unit, configured to perform a first process on the target RGBIR image using a first neural network to obtain a target IR image; and / or perform a second process on the target RGBIR image using a second neural network to obtain a target RGB image; The first processing includes: eliminating the visible light component in the pixel value corresponding to the IR channel pixel point in the target RGBIR image, and predicting the pixel value of the color channel pixel point in the IR channel, and the second processing includes: for each color channel, eliminating the near infrared light component in the pixel value corresponding to the color channel pixel point in the target RGBIR image, and predicting the pixel value of the pixel point other than the color channel pixel point in the color channel; A first training unit is configured to obtain a plurality of first training image pairs before obtaining a target RGBIR image, wherein the first training image pairs include: a first RGBIR training image and a label image corresponding to the first RGBIR training image, wherein the label image is an IR image without visible light interference, and the label image is obtained by performing a first integral operation on a multispectral image corresponding to the first RGBIR training image; and train a first neural network using the plurality of first training image pairs; The second training unit is configured to obtain a plurality of second training image pairs before obtaining the target RGBIR image, the second training image pairs comprising: a second RGBIR training image and a label image corresponding to the second RGBIR training image, wherein the label image corresponding to the second RGBIR training image is an RGB image without near-infrared light interference, and the label image corresponding to the second RGBIR training image is obtained by performing a second integral operation on the multispectral image corresponding to the second RGBIR training image; and train the second neural network using the plurality of second training image pairs.
9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7. 10 . A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to claim 1 .
