Image processing method and device, equipment, storage medium and program product

Abstract

The invention discloses an image processing method and device, equipment, a storage medium and a program product, the method is applied to electronic equipment, the electronic equipment comprises a first image sensor, the first image sensor is a multispectral sensor, and the method specifically comprises the steps of obtaining a multispectral image collected by the first image sensor; and performing first image processing on the multispectral image to obtain a first image in a first color space, the first image processing including color space conversion processing, and the number of channels of the multispectral image being greater than the number of channels of the first image.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Common RGB sensors can capture data of three channels: red (R), green (G), and blue (B), while multispectral sensors can capture more color channel data corresponding to different bands. In these different bands, substances and materials have different ways of reflecting, absorbing, or transmitting light. Therefore, compared with RGB sensors, multispectral sensors can provide richer information to help comprehensively understand the observed object. Multispectral cameras are widely used in multiple fields such as remote sensing, environmental monitoring, and medical treatment. In recent years, multispectral sensors have begun to be used in the field of photography to assist in improving the accuracy of colors.

[0003] Although the improvement of the spectral resolution of multispectral images can obtain more spectral information in multiple bands and enhance the analysis ability of the spectral characteristics of objects, this process is often accompanied by the sacrifice of spatial resolution, thereby affecting image clarity and detail expressiveness. Summary of the Invention

[0004] Embodiments of the present application are expected to provide an image processing method, apparatus, device, storage medium, and program product.

[0005] The technical solution of the present application is implemented as follows:

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device includes a first image sensor, and the first image sensor is a multispectral sensor. The method includes:

[0007] Obtain a multispectral image collected by the first image sensor;

[0008] Perform first image processing on the multispectral image to obtain a first image in a first color space. The first image processing includes color space conversion processing, and the number of channels of the multispectral image is greater than the number of channels of the first image.

[0009] In a second aspect, an image processing apparatus is provided, which is applied to an electronic device. The electronic device includes a first image sensor, and the first image sensor is a multispectral sensor. The apparatus includes:

[0010] An obtaining unit, configured to obtain a multispectral image collected by the first image sensor;

[0011] A first processing unit, configured to perform first image processing on the multispectral image to obtain a first image in a first color space, where the first image processing includes color space conversion processing, and the number of channels of the multispectral image is greater than the number of channels of the first image.

[0012] In a third aspect, an electronic device is provided, which is characterized in that the electronic device includes: a first image sensor, a processor, and a memory configured to store a computer program that can run on the processor.

[0013] Wherein, the first image sensor is a multispectral sensor, and when the processor is configured to run the computer program, it executes the steps of the foregoing method.

[0014] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the foregoing method are implemented.

[0015] In a fifth aspect, a computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the foregoing method are implemented.

[0016] In the embodiments of the present application, an image processing method, apparatus, device, storage medium, and program product are provided. The method is applied to an electronic device, and the electronic device includes a first image sensor, and the first image sensor is a multispectral sensor. The method specifically includes: acquiring a multispectral image collected by the first image sensor; performing first image processing on the multispectral image to obtain a first image in a first color space, where the first image processing includes color space conversion processing, and the number of channels of the multispectral image is greater than the number of channels of the first image. In this way, by using the rich spectral information of the multispectral image and converting the multispectral image into a traditional color space, the color reproduction accuracy and image resolution can be improved, thereby improving the imaging quality.

[0017] Furthermore, the converted first image can directly apply mature image processing methods for this first color space, thereby greatly reducing the complexity and difficulty of multispectral image processing, making the application of multispectral images more extensive and convenient. Description of the Drawings

[0018] Figure 1 It is a schematic diagram of the spectral response curve of human eye cone cells in an embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of the spectral response curve of an RGB sensor in an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of the spectral response curve of a multispectral sensor in an embodiment of the present application;

[0021] Figure 4 It is the first flow schematic diagram of the image processing method in the embodiment of the present application;

[0022] Figure 5 It is the flow schematic diagram of the training data set determination method in the embodiment of the present application;

[0023] Figure 6 It is the schematic diagram of the pixel arrangement mode of the multispectral sensor in the embodiment of the present application;

[0024] Figure 7 It is the first structural schematic diagram of the neural network in the embodiment of the present application;

[0025] Figure 8 It is the second structural schematic diagram of the neural network in the embodiment of the present application;

[0026] Figure 9 It is the second flow schematic diagram of the image processing method in the embodiment of the present application;

[0027] Figure 10 It is the composition structural schematic diagram of the image processing device in the embodiment of the present application;

[0028] Figure 11 It is the composition structural schematic diagram of the electronic device in the embodiment of the present application. Detailed implementation manners

[0029] In order to be able to understand the features and technical content of the embodiments of the present application in more detail, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are only for reference and explanation, and are not used to limit the embodiments of the present application.

[0030] The visual principle of the human eye is based on the reception of the spectral band by the cone cells in the human eye and the generation of nerve signals. There are three types of cone cells in the human eye, which are sensitive to light in the long wavelength band, medium wavelength band, and short wavelength band respectively. Their response peaks correspond to the blue, green, and red spectral regions, and their response characteristics can be characterized by a spectral response curve, as Figure 1 shown.

[0031] Ordinary color cameras, such as RGB cameras, work on the principle that to a certain extent simulates the visual principle of the human eye. The red sub-pixels, green sub-pixels, and blue sub-pixels in the camera are designed to respond to light in the long wavelength band, medium wavelength band, and short wavelength band respectively, and generate corresponding electrical signals. The response characteristics of these electrical signals can also be described by a spectral response curve, Figure 2 which shows the spectral response curves of the three colors of red, green, and blue. The spectral range of the RGB camera is (400 - 700nm), and the number of filters is 3.

[0032] A multispectral camera can acquire spectral information in multiple bands (usually more than three) compared to an RGB camera. Figure 3 The spectral response curve of a multispectral camera is shown. The spectral range of the multispectral camera (350 - 1000 nm) has a larger number of filters. The breadth and fineness of the light signals captured by the multispectral camera are higher than those of the RGB camera. The core features and challenges of the multispectral camera can be summarized as follows:

[0033] The core features of the multispectral camera include:

[0034] (1) High spectral resolution: Through multi - band sampling (usually more than 10 spectral channels), it can effectively distinguish subtle color differences that are indistinguishable to the human eye and RGB cameras. For example, fabrics with similar colors may appear the same in RGB imaging, but multispectral data can restore the true color differences.

[0035] (2) Wide spectral band coverage: It supports multi - spectral imaging in ultraviolet, visible, infrared, etc., breaking through the limitation of the visible light range of 380 - 750 nm of the human eye, and can obtain spectral information on the characteristics of substances.

[0036] (3) Precise color restoration: Based on high - dimensional spectral data, through optimizing the spectral reconstruction algorithm, it can achieve a color reproduction effect that is more in line with human eye perception.

[0037] Currently, the technical challenges faced by multispectral cameras include:

[0038] (1) System compatibility obstacles: The traditional RGB three - channel image processing system (including ISP chips, codecs, display devices) is difficult to adapt to multispectral data.

[0039] (2) Spatial resolution loss: When the total number of pixels is fixed, the spatial resolution of a multispectral camera is inversely proportional to the number of spectral channels. For example, for an RGB sensor with 300×300 sub - pixels, with R, G, and B forming a pixel unit, the number of pixel units is 300×300 / 3 = 30000. For a multispectral sensor with the same number of sub - pixels, if its pixel unit contains 10 channels, the number of pixel units is 300×300 / 10 = 9000, which results in a significantly lower imaging resolution of the multispectral sensor compared to an RGB camera of the same specification.

[0040] In view of this, in the embodiments of the present application, an image processing method, device, equipment, storage medium, and program product are provided. The method is applied to an electronic device, and the electronic device at least includes a first image sensor, and the first image sensor is a multispectral sensor for collecting multispectral images. Utilizing the rich spectral information of the multispectral images, by converting the multispectral images into a traditional color space, it is possible to improve the color restoration accuracy and image resolution, thereby improving the imaging quality.

[0041] Furthermore, the converted first image can directly apply mature image processing methods for this first color space, thereby greatly reducing the difficulty of multi-spectral image processing and making the application of multi-spectral images more extensive and convenient.

[0042] Figure 4 This is the first process schematic diagram of the image processing method in the embodiments of this application. As Figure 4 shown, this method may specifically include:

[0043] Step 401: Obtain a multi-spectral image collected by a first image sensor;

[0044] Step 402: Perform first image processing on the multi-spectral image to obtain a first image in a first color space. The first image processing includes color space conversion processing, and the number of channels of the multi-spectral image is greater than the number of channels of the first image.

[0045] In some embodiments, based on the color matching function of the first color space, perform color space conversion processing on the multi-spectral image to obtain a first image.

[0046] Exemplarily, the first color space includes but is not limited to one of the following: RGB, XYZ, Lab, YUV, etc.

[0047] Exemplarily, taking CIE RGB as an example for the first color space, the CIE RGB color matching function is as follows:

[0048]

[0049] where I(λ) is the spectral data of each pixel, are the tristimulus values, and R, G, and B are the three color components of the RGB color space respectively.

[0050] Exemplarily. The data of the CIE RGB color matching function, that is, can exist in tabular form, with three coefficients corresponding to each wavelength. However, in actual operation, the integration operation can be replaced by summation, especially when the spectral data is discretely sampled. The CIE RGB color matching function can be transformed into:

[0051]

[0052]

[0053] Exemplarily, the color space conversion processing can be as follows:

[0054] 1. Determine the wavelength range and interval of the spectral data I(λ), such as from 380 nm to 780 nm with a 1 nm interval. Assume that the spectral data is already discretized, and the intensity corresponding to each wavelength λi is I(λi).

[0055] 2. Obtain the data of the CIE RGB color matching functions, that is, the three tristimulus values

[0056] 3. For each wavelength λi, multiply I(λi) by the corresponding and then multiply by the wavelength interval Δλ (for example, if it is 1 nm, Δλ = 1), and then sum over all wavelengths to obtain the three color components R, G, and B.

[0057] In some embodiments, the first image processing further includes demosaicing processing and / or noise reduction processing.

[0058] Exemplarily, perform demosaicing processing on the multispectral image to obtain a second image; perform color space conversion processing on the second image to obtain the first image in the first color space.

[0059] Exemplarily, perform noise reduction processing on the multispectral image to obtain a third image; perform color space conversion processing on the third image to obtain the first image in the first color space.

[0060] Exemplarily, perform demosaicing processing on the multispectral image to obtain a second image; perform noise reduction processing on the second image to obtain a third image; perform color space conversion processing on the third image to obtain the first image in the first color space.

[0061] Exemplarily, perform noise reduction processing on the multispectral image to obtain a second image; perform demosaicing processing on the second image to obtain a third image; perform color space conversion processing on the third image to obtain the first image in the first color space.

[0062] If each pixel of the multispectral sensor only contains single-channel spectral information, the mosaic-like artifacts in the original data can be eliminated through demosaicing. For example, the demosaicing of the spectral image can include: based on the spectral correlation of neighboring pixels, using adaptive interpolation (bilinear / edge-guided / deep learning) to complete the missing channel data; through spectral response function calibration, eliminating channel crosstalk and spectral distortion caused by interpolation, reconstructing the complete spectral image, and through the color information fusion of neighboring pixels, restoring the true color representation of high-frequency details (such as textures and edges) to enhance the visual coherence of the image. Exemplarily, assume that the size of the original spectral image output by the multispectral sensor is H×W×1, and the size of the spectral image after demosaicing is H×W×M, where M is the number of channels of the multispectral image.

[0063] Through noise reduction processing, the sensor noise and interference introduced by the transmission link can be reduced or eliminated, improving the signal-to-noise ratio.

[0064] In some embodiments, the performing first image processing on the multispectral image to obtain a first image in a first color space may include: obtaining the luminance information of the multispectral image; performing first image processing on the multispectral image based on the luminance information of the multispectral image to obtain a first image in a first color space.

[0065] Exemplarily, based on the luminance information of the multispectral image and a mapping relationship, determining a first image processing strategy; based on the first image processing strategy, performing first image processing on the multispectral image to obtain a first image in a first color space, and the mapping relationship includes one or more mapping relationships between luminance information and image processing strategies. For example, the luminance information can be the luminance information of the current shooting environment or the average luminance of the multispectral image. By adaptively adjusting the image processing strategy according to the luminance, the image processing effect can be improved.

[0066] Exemplarily, performing first image processing on the multispectral image based on the luminance information of the multispectral image to obtain a first image in a first color space. For example, the luminance information can be local or pixel-level luminance. Executing appropriate image processing strategies for image regions with different luminances is beneficial to improving the image processing accuracy.

[0067] In some embodiments, the performing first image processing on the multispectral image to obtain a first image in a first color space may include: using a neural network to perform first image processing on the multispectral image to obtain the first image in the first color space.

[0068] Exemplarily, a first neural network is determined based on the luminance information and mapping relationship of the multispectral image; the multispectral image is input into the first neural network for first image processing to obtain a first image in a first color space, and the mapping relationship includes the mapping relationship between one or more luminance information and the neural network.

[0069] Exemplarily, the luminance information of the multispectral image and the multispectral image are input into a neural network for first image processing to obtain a first image in a first color space.

[0070] In some embodiments, the training dataset of the neural network includes sample images and ground truth images pairs, the sample images and ground truth images are generated based on hyperspectral images, the number of channels of the hyperspectral image is greater than or equal to the number of channels of the multispectral image, and each pixel in the hyperspectral image includes spectral intensities of multiple channels.

[0071] The hyperspectral image can be obtained from a specific database or high-quality spectral data can be collected by a high-precision device in a specific environment, and it contains more channels than those captured by a multispectral sensor.

[0072] The sample image, as an output image of a multispectral sensor, is a multispectral image, that is, the number of channels of the sample image is the same as that of the first image. The sample image

[0073] In some embodiments, based on the spectral response function of the multispectral sensor and the hyperspectral image, a fourth image collected by the multispectral sensor is determined; one or more noises are added to the fourth image to obtain a fifth image of the multispectral sensor; based on the fifth image of the multispectral sensor, the sample image is determined.

[0074] For example, single-channel data extraction is performed on the fifth image to obtain the sample image. Another example is to use the fifth image as the sample image.

[0075] In some embodiments, the ground truth image includes a first ground truth image, and the first ground truth image is obtained by performing color space conversion processing on the hyperspectral image based on the color matching function of the first color space.

[0076] Exemplarily, the neural network training process may include: inputting the sample image into the neural network to obtain an output image; calculating a loss function based on the output image and the first ground truth image; optimizing the neural network parameters to minimize the loss function to obtain a trained neural network.

[0077] Exemplarily, the neural network training process may include: inputting a sample image into the neural network to obtain a first output image and a second output image; calculating a first loss function based on the first output image and a first ground truth image, and calculating a second loss function based on the second output image and a second ground truth image; performing weighted fusion on the first loss function and the second loss function to obtain a total loss function, and optimizing the neural network parameters to minimize the total loss function to obtain a trained neural network. The first output image is the image finally output by the neural network, the second output image is one or more images output in the intermediate process of the neural network, and the second ground truth image includes one or more ground truth images corresponding to the intermediate process of the neural network.

[0078] In some embodiments, the ground truth image further includes a second ground truth image and / or a third ground truth image.

[0079] Exemplarily, the second ground truth image may be a fourth image determined based on the spectral response function of the multispectral sensor and the hyperspectral image; the second ground truth image may be a fifth image obtained by adding one or more types of noise to the fourth image.

[0080] Figure 5 Schematic diagram of the process for determining the training dataset in the embodiments of the present application, as Figure 5 shown, the method for determining the training dataset may specifically include:

[0081] Step 501: Obtain a hyperspectral image / image data;

[0082] Step 502: Based on the color matching function, convert the hyperspectral image into a three-channel CIE RGB color space to obtain an RGB image / image data;

[0083] Exemplarily, the number of color channels of the multispectral sensor is 8, that is, the number of channels M of the multispectral image is 8. The hyperspectral image reflects the spectral intensity of each pixel in a wider spectral range (such as 380 nm to 780 nm) at intervals of 5 nm, and the number of color channels is 80.

[0084] Using the color matching function, the spectral data corresponding to the hyperspectral image can be converted into a three-channel standard color space. For example, if the spectral data of a certain pixel in the hyperspectral image is I(λ), the CIE RGB color matching function can be used, that is are the tristimulus values to obtain the RGB three-color values of this pixel in the CIE RGB space.

[0085] Step 503: Based on the spectral response function of the multispectral sensor, convert the hyperspectral image into the spectral data of each channel of the multispectral sensor, which may be the ideal Raw image output by the multispectral sensor;

[0086] For example, based on the spectral response function of the multispectral sensor, the value of each channel of the multispectral sensor can be obtained. Assume that the multispectral sensor has M color channels, and the spectral response function of the i-th channel is SRF i , then the sensor value of the i-th channel is,

[0087]

[0088] Step 504: adding noise to the multispectral image according to the noise model;

[0089] The above multispectral sensor values ​​are ideal, but the actual sensor has noise interference such as Poisson noise and Gaussian noise. It is necessary to add noise to the ideal values ​​through the noise model to simulate the real response data of the multispectral sensor.

[0090] In some embodiments, the ideal Raw image can be converted into a Raw image with noise according to the relationship between the brightness and noise model. noise .

[0091] For example, based on the physical model of sensor noise, the noise image Img noise With the denoised image Img clean It conforms to the following formula,

[0092]

[0093] Among them, Poisson() represents the Poisson distribution function, N() represents the Gaussian distribution function, k, σ 2 is the noise model parameter related to the ambient brightness.

[0094] k, σ 2 The specific method is to take multiple images with a fixed multispectral camera under a certain ambient brightness. These multiple images can be regarded as Img with noise. noise ; Superimpose these multiple images to obtain Img clean ; Substitute into the above formula, and fit to get k and σ at this brightness 2 By conducting the above test under various ambient brightness conditions, k and σ can be established. 2 The mapping table or mapping function of brightness is as follows: Y represents brightness,

[0095] k = func1(Y)

[0096] σ 2 =func2(Y)

[0097] According to the CIE standard luminosity function, the image brightness Y can be calculated based on the spectral distribution I(λ) of the hyperspectral image.

[0098]

[0099] In summary, after establishing the noise model of the multispectral sensor, it is possible to convert from the ideal Raw to the Raw. noise , and the calculation formula can be:

[0100]

[0101] Step 505: Extract single-channel data to obtain the simulated Raw data of the multispectral sensor

[0102] According to the spectral data I(λ) of a certain pixel, the response values Raw of M channels of the multispectral sensor are simulated and calculated. noise . Raw noise has the size of HxWxM. In fact, each sub-pixel of the multispectral sensor can only capture the data of one channel. According to the pixel arrangement, the Raw with the size of HxWxM noise should be extracted into the Raw with the size of HxWx1 mosaic , and each pixel only retains the value of the corresponding channel. As Figure 6 shown, each pixel unit of the multispectral sensor in the figure contains spectral data of 8 channels, and the size of Raw noise is HxWx8. Assuming that the upper-left sub-pixel can only receive the red channel and its left sub-pixel can only receive the green channel, data needs to be extracted so that each sub-pixel only retains the corresponding 1-channel data.

[0103] After the above steps, based on the hyperspectral image, the corresponding three-channel ideal RGB image, ideal Raw, noise Raw noise and the Raw that simulates the real output of the multispectral sensor mosaic。 are established to form a training dataset.

[0104] Figure 7 is the first structural schematic diagram of the neural network in the embodiment of the present application. As Figure 7 shown, the neural network can be designed with a single input and a single output. The input is Raw mosaic , and the true value is the RGB image of three channels. The loss function calculated by using the output image of the neural network and the true value image is used to update the neural network.

[0105] Figure 8 is the second structural schematic diagram of the neural network in the embodiment of the present application. As Figure 8 shown, the neural network can also be designed with a single input, multiple outputs, and multiple true values. It can be designed as a series structure of multiple modules, and multiple modules are respectively used to implement functions such as noise reduction, demosaicing, and color space conversion. For example, the input of module one is Raw mosaic , and the output value of module one and the true value Raw noiseCalculate the first loss function so that Module 1 can implement the demosaicing function; Module 2 uses the output of Module 1 as the input value, and calculates the second loss function using the output value of Module 2 and the true value Raw, so that Module 2 can implement the noise reduction function; Module 3 uses the output of Module 2 as the input value, and calculates the third loss function using the output value of Module 3 and the true value RGB, so that Module 3 can implement the color mapping from multi-channel color values to three-channel color values. The weighted sum of the loss functions of the above three modules is used as the total loss function for training the neural network.

[0106] The above neural network has the functions of noise reduction, demosaicing, and color space conversion in this solution. It maps the original multi-spectral image data output by the multi-spectral sensor into RGB image data, obtaining higher color discrimination and a color performance closer to the human eye than RGB sensors, and the image spatial resolution can be consistent with RGB sensors. When training the neural network, a training dataset can be constructed based on homologous spectral data, avoiding the registration deviation of multi-device acquisition and obtaining a high-quality dataset.

[0107] To better illustrate the purpose of this application, based on the above embodiments of this application, further examples are given, such as Figure 9 As shown, the method may specifically include:

[0108] Step 901: Obtain a multi-spectral image collected by a first image sensor;

[0109] Step 902: Perform a first image processing on the multi-spectral image to obtain a first image in a first color space. The first image processing includes color space conversion processing, and the number of channels of the multi-spectral image is greater than the number of channels of the first image;

[0110] Step 903: Perform a second image processing on the first image to obtain a target image. The second image processing includes one or more image processings for the image in the first color space.

[0111] The second image processing can be implemented by a traditional color space body (such as RGB, XYZ, Lab, YUV, etc.) image processing system, enabling the image processing system to have the ability to process multi-spectral images, thereby achieving wide compatibility.

[0112] In some embodiments, the second image processing includes, but is not limited to, one or more of noise reduction, white balance adjustment, color interpolation, color correction, gamma correction, color space conversion, etc.

[0113] In some embodiments, the electronic device further includes a second image sensor, and the method further includes: obtaining a sixth image in the first color space collected by the second image sensor; determining a target image based on the first image and the sixth image.

[0114] The second image sensor includes, but is not limited to, a multispectral image sensor, an RGB image sensor, etc.

[0115] In some embodiments, the second image processing includes, but is not limited to, one or more of registration, calibration, fusion, stitching, etc.

[0116] By adopting the above technical solution, by converting the multispectral image into a traditional color space, the converted first image can directly apply mature image processing methods for the first color space, thereby greatly reducing the complexity and difficulty of multispectral image processing, and making the application of multispectral images more extensive and convenient.

[0117] To implement the method of the embodiments of the present application, based on the same inventive concept, the embodiments of the present application also provide an image processing apparatus, which is applied to an electronic device. The electronic device includes a first image sensor, and the first image sensor is a multispectral sensor, such as Figure 10 As shown, the image processing apparatus 100 includes:

[0118] An obtaining unit 1001, configured to obtain a multispectral image collected by the first image sensor;

[0119] A processing unit 1002, configured to perform first image processing on the multispectral image to obtain a first image in the first color space. The first image processing includes color space conversion processing, and the number of channels of the multispectral image is greater than the number of channels of the first image.

[0120] In some embodiments, the first image processing further includes demosaicing processing and / or noise reduction processing.

[0121] In some embodiments, the processing unit 1002 is configured to perform demosaicing processing on the multispectral image to obtain a second image; perform noise reduction processing on the second image to obtain a third image; and perform color space conversion processing on the third image to obtain the first image in the first color space.

[0122] In some embodiments, the processing unit 1002 is configured to perform demosaicing processing on the multispectral image to obtain a second image; and perform color space conversion processing on the second image to obtain the first image in the first color space.

[0123] In some embodiments, the processing unit 1002 is configured to perform noise reduction processing on the multi-spectral image to obtain a third image, and perform color space conversion processing on the third image to obtain the first image in the first color space.

[0124] In some embodiments, the processing unit 1002 is configured to obtain the luminance information of the multi-spectral image, and perform first image processing on the multi-spectral image based on the luminance information of the multi-spectral image to obtain the first image in the first color space.

[0125] In some embodiments, the processing unit 1002 performs first image processing on the multi-spectral image by using a neural network to obtain the first image in the first color space.

[0126] In some embodiments, the training data set of the neural network includes a sample image and a ground truth image pair. The sample image and the ground truth image are generated based on a hyperspectral image. The number of channels of the hyperspectral image is greater than or equal to the number of channels of the multi-spectral image. Each pixel in the hyperspectral image includes spectral intensities of multiple channels.

[0127] In some embodiments, based on the spectral response function of the multi-spectral sensor and the hyperspectral image, a fourth image collected by the multi-spectral sensor is determined; one or more types of noise are added to the fourth image to obtain a fifth image of the multi-spectral sensor; based on the fifth image of the multi-spectral sensor, the sample image is determined.

[0128] In some embodiments, single-channel data extraction is performed on the fifth image to obtain the sample image.

[0129] In some embodiments, the ground truth image includes a first ground truth image, and the first ground truth image is obtained by performing color space conversion processing on the hyperspectral image based on the color matching function of the first color space.

[0130] In some embodiments, the ground truth image further includes a second ground truth image and / or a third ground truth image. The second ground truth image is a fourth image determined based on the spectral response function of the multi-spectral sensor and the hyperspectral image; the second ground truth image is a fifth image obtained after adding one or more types of noise to the fourth image.

[0131] In some embodiments, the processing unit 1002 is further configured to perform second image processing on the first image to obtain a target image, and the second image processing includes one or more types of image processing on the image in the first color space.

[0132] In some embodiments, the electronic device further includes a second image sensor. The processing unit 1002 is further configured to obtain a sixth image in the first color space collected by the second image sensor, and determine a target image based on the first image and the sixth image.

[0133] In practical applications, the above device can be an electronic device or a chip applied to an electronic device. In this application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the image processing method provided in any of the foregoing embodiments. And the technical effects of the technical solutions of the device can refer to the technical effects of the corresponding technical solutions in the image processing method, which will not be elaborated herein one by one.

[0134] Based on the hardware implementation of each unit in the above image processing device, an embodiment of the present application further provides an electronic device, as Figure 11 shown. The electronic device 110 includes a processor 1101 and a memory 1102 configured to store a computer program that can run on the processor.

[0135] Wherein, when the processor 1101 is configured to run the computer program, it executes the method steps in the foregoing embodiments.

[0136] Of course, in practical applications, as Figure 11 shown, each component in the electronic device 110 is coupled together through a bus system 1103. It can be understood that the bus system 1103 is used to realize the connection and communication between these components. The bus system 1103 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all kinds of buses are labeled as the bus system 1103 in the figure.

[0137] In practical applications, the above processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic devices for implementing the functions of the above processor can be other, and the embodiments of the present application do not make specific limitations.

[0138] The above-mentioned memory may be a volatile memory, such as a random-access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or a combination of the above types of memories, and provides instructions and data to the processor.

[0139] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the foregoing method.

[0140] The embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the embodiments of the present application are implemented.

[0141] Optionally, the computer program product can be applied to the electronic device in the embodiment of the present application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in each method of the embodiment of the present application. For the sake of brevity, it will not be elaborated here.

[0142] The embodiment of the present application further provides a computer program.

[0143] Optionally, the computer program can be applied to the electronic device in the embodiment of the present application. When the computer program runs on the computer, the computer is caused to execute the corresponding processes implemented by the electronic device in each method of the embodiment of the present application. For the sake of brevity, it will not be elaborated here.

[0144] It should be understood that in the embodiment of the present application, regarding data such as user information, when the embodiment of the present application is applied to a specific product or technology, user permission or consent is required, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0145] It should be understood that the terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The expressions "have", "may have", "include", "contain", "may include", and "may contain" used in this application can be used herein to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations, or components), but do not exclude the presence of additional features.

[0146] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and do not necessarily describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0147] Among the technical solutions described in the embodiments of this application, they can be combined arbitrarily without conflict.

[0148] In several embodiments provided in this application, it should be understood that the disclosed methods, devices, and equipment can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0149] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in the embodiments of this application can be all integrated in a processing unit, or each unit can be separately used as a unit, or two or more units can be integrated in one unit; the above integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0151] As described above, it is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device includes a first image sensor, and the first image sensor is a multispectral sensor. The method includes: Obtain a multispectral image collected by the first image sensor; Perform first image processing on the multispectral image to obtain a first image in a first color space. The first image processing includes color space conversion processing, and the number of channels of the multispectral image is greater than the number of channels of the first image.

2. The method according to claim 1, wherein The first image processing further includes demosaicing processing and / or noise reduction processing.

3. The method according to claim 2, wherein The performing first image processing on the multispectral image to obtain a first image in a first color space includes: Perform demosaicing processing on the multispectral image to obtain a second image; Perform noise reduction processing on the second image to obtain a third image; Perform color space conversion processing on the third image to obtain the first image in the first color space.

4. The method according to claim 1, wherein The performing first image processing on the multispectral image to obtain a first image in a first color space includes: Obtain the luminance information of the multispectral image; Based on the luminance information of the multispectral image, perform first image processing on the multispectral image to obtain a first image in a first color space.

5. The method according to any one of claims 1-4, characterized in that, The performing first image processing on the multispectral image to obtain a first image in a first color space includes: Use a neural network to perform first image processing on the multispectral image to obtain the first image in the first color space.

6. The method according to claim 5, characterized in that, The training data set of the neural network includes a sample image and a ground truth image pair. The sample image and the ground truth image are generated based on a hyperspectral image. The number of channels of the hyperspectral image is greater than or equal to the number of channels of the multispectral image, and each pixel in the hyperspectral image includes spectral intensities of multiple channels.

7. The method according to claim 6, characterized in that, The method further includes: Based on the spectral response function of the multispectral sensor and the hyperspectral image, determine a fourth image collected by the multispectral sensor; Add one or more types of noise to the fourth image to obtain a fifth image of the multispectral sensor; Based on the fifth image of the multispectral sensor, determine the sample image.

8. The method according to claim 7, wherein The method further includes: Perform single-channel data extraction on the fifth image to obtain the sample image.

9. The method according to claim 6, characterized in that, The ground truth image includes a first ground truth image, and the first ground truth image is obtained by performing color space conversion processing on the hyperspectral image based on the color matching function in the first color space.

10. The method according to claim 9, wherein The ground truth image further includes a second ground truth image and / or a third ground truth image. The second ground truth image is a fourth image determined based on the spectral response function of the multispectral sensor and the hyperspectral image; The second ground truth image is a fifth image obtained after adding one or more types of noise to the fourth image.

11. The method according to claim 1, characterized in that, The method further includes: Perform second image processing on the first image to obtain a target image. The second image processing includes one or more types of image processing for the image in the first color space.

12. The method according to claim 1, characterized in that The electronic device further includes a second image sensor. The method further includes: Obtain a sixth image in the first color space collected by the second image sensor; Determine a target image based on the first image and the sixth image.

13. An image processing apparatus, characterized in that, Applied to an electronic device, the electronic device includes a first image sensor, and the first image sensor is a multispectral sensor. The device includes: An acquisition unit, configured to acquire a multispectral image collected by the first image sensor; A processing unit, configured to perform a first image processing on the multispectral image to obtain a first image in a first color space. The first image processing includes a color space conversion process, and the number of channels of the multispectral image is greater than the number of channels of the first image.

14. An electronic device, characterized in that, The electronic device includes: a first image sensor, a processor, and a memory configured to store a computer program that can run on the processor, wherein the first image sensor is a multispectral sensor, and when the processor is configured to run the computer program, it executes the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 12.

Images