Image processing method and apparatus

Abstract

The present application provides an image processing method and apparatus, which are used for assisting in the enhancement of texture and details of an input image by utilizing a hyperspectral image, so that a denoising effect for the input image is achieved, and an output image having a high signal-to-noise ratio is obtained. The method comprises: first, acquiring an input image and a hyperspectral image, the input image being an image acquired by an image sensor, and the spectral channels of the hyperspectral image not being completely identical to the spectral channels of the input image, e.g., the number of channels of the hyperspectral image being greater than the number of channels of the input image, or the hyperspectral image comprising a channel not contained in the input image, etc., which can be understood as the hyperspectral image being able to comprise more abundant information or more accurate information compared to the input image; then, extracting a feature from the input image, to obtain a first feature, and obtaining a second feature on the basis of a feature extracted from the hyperspectral image; and fusing the first feature and the second feature, to obtain a fused feature, and then, on the basis of the fused feature, performing recovery, to obtain an output image.

Description

Image processing method and device

[0001] The present application claims priority to the Chinese patent application No. 202410548810.4, filed on April 30, 2024, and entitled “Image processing method and device”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of computer vision, and in particular, to an image processing method and device. BACKGROUND

[0003] An image sensor is a sensor used for collecting images in an electronic device. The imaging process of the image sensor is usually affected by noise, which comes from the imaging module or circuit, etc. The noise is usually manifested as noise points, blurring or artifacts in the image, which will cause the quality of the image to decrease. For example, in some scenarios, such as dark light scenarios or strong light scenarios, there may be a loss of details or textures of the image. Image denoising methods aim to remove noise in the image and preserve the original content of the image. For example, a commonly used image denoising method can use information from other sensors other than the image sensor to assist in image denoising. However, the existing auxiliary denoising scheme has limited denoising effect on the image, and therefore, how to improve the image denoising effect becomes a problem to be solved. SUMMARY

[0004] The present application provides an image processing method and device for enhancing the texture and details of an input image with the aid of a hyperspectral image, so as to achieve the denoising effect of the input image and obtain an output image with high signal-to-noise ratio.

[0005] Therefore, in a first aspect, the present application provides an image processing method, which comprises: first, obtaining an input image and a hyperspectral image, the input image being an image collected by an image sensor, and the spectral channels of the hyperspectral image being different from those of the input image, such as the number of channels of the hyperspectral image being greater than that of the input image, or the hyperspectral image including channels not contained in the input image, etc. It can be understood that the hyperspectral image can include more abundant information or more accurate information than the input image; then extracting features from the input image to obtain first features, and extracting features from the hyperspectral image to obtain second features; fusing the first features and the second features to obtain fused features, and then restoring the output image according to the fused features.

[0006] In the embodiments of the present application, the richer or more accurate information included in the hyperspectral image can be used to assist in the enhancement or denoising of the input image, so as to enhance the texture or details of the input image, and reduce the noise in the input image, thereby obtaining an output image with a higher signal-to-noise ratio.

[0007] In a possible implementation, the fusing of the first feature and the second feature to obtain the fused feature can include: fusing the first feature and the second feature based on an attention mechanism to obtain the fused feature.

[0008] In the embodiments of the present application, the features extracted from the input image and the hyperspectral image can be fused based on the attention mechanism, so that the attention mechanism can be used to pay attention to the similarity between the features, and the similarity between the features can be used to assist in enhancing the features extracted from the input image, thereby achieving the enhancement effect of the input image, reducing the noise in the input image, and obtaining an output image with a higher signal-to-noise ratio.

[0009] In a possible implementation, the fusing of the first feature and the second feature to obtain the fused feature can further include: obtaining a historical frame image, the historical frame image being a previous frame image of the input image, for example, the input image being one frame of video data, and the historical frame image being a previous frame image of the input image in the video data; and extracting a feature from the historical frame image to obtain a third feature, and fusing the first feature, the second feature and the third feature to obtain the fused feature. In the embodiments of the present application, the current frame can be further enhanced in combination with the historical frame image, thereby further reducing the noise in the current frame image.

[0010] In a possible implementation, the obtaining of the second feature according to the feature extracted from the hyperspectral image can include: screening information of a spectral channel from the hyperspectral image to obtain a spectral channel reconstruction image; and obtaining the second feature according to the feature extracted from the spectral channel reconstruction image. In the embodiments of the present application, the feature of the required channel can be screened from the hyperspectral image, so that the spectral channel reconstruction image is reconstructed, and the dimension of the feature extracted subsequently can be aligned with the dimension of the input image, so as to facilitate the enhancement of the feature of the input image based on the feature of the required channel subsequently.

[0011] In a possible implementation, the screening of the information of the preset spectral channel from the hyperspectral image to obtain the spectral channel reconstruction image can include: extracting a feature from the hyperspectral image to obtain a third feature; obtaining a dynamic attention coefficient and a spectral feature according to the third feature based on an attention mechanism; and fusing the dynamic attention coefficient, the spectral feature and a preset static attention coefficient to obtain the spectral channel reconstruction image.

[0012] In the embodiment of the present application, when the hyperspectral image is reconstructed in the channel, the attention mechanism can be used to reconstruct the required spectral features based on the similarity between the features, so that the features in the obtained channel are more accurate and the data error can be avoided.

[0013] In a possible implementation, the foregoing method of extracting the features in the reconstructed image according to the spectral channel includes: performing at least one down-sampling on the reconstructed image according to the spectral channel to obtain at least one down-sampled hyperspectral image; and extracting features from the at least one down-sampled hyperspectral image to obtain at least one scale feature, and the second feature includes the at least one scale feature.

[0014] In the embodiment of the present application, in the process of extracting the second feature, one or more down-samplings can be performed to obtain features of different scales, so that the features in the input image can be enhanced from different scales in the subsequent process, the image can be enhanced or de-noised from different scales, the fused feature with less noise can be obtained, and the output image with less noise can be obtained.

[0015] In a possible implementation, the foregoing method of fusing the first feature and the second feature to obtain the fused feature can further include: similarly to the foregoing down-sampling of the reconstructed image according to the spectral channel, at least one down-sampling can be performed on the first feature to obtain at least one down-sampled feature, so as to align the scales, and the scales of the at least one down-sampled feature correspond to the scales of the at least one scale feature one by one; and the at least one down-sampled feature is fused with the corresponding scale feature to obtain at least one scale fused feature, and then the at least one scale fused feature is fused to obtain the fused feature. Therefore, in the embodiment of the present application, the first feature and the second feature can be fused from different scales, so that the enhancement or de-noising effect can be achieved in each scale, and the noise in the obtained fused feature can be reduced.

[0016] In a possible implementation, the foregoing method further includes: performing aberration estimation according to the hyperspectral image to obtain an aberration estimation image, the aberration estimation image being used to represent the error of a pixel point in the input image; and performing recovery on the fused feature according to the aberration estimation image to obtain the output image.

[0017] In the embodiment of the present application, for the possible aberration, the aberration estimation can be performed based on the hyperspectral image, so that the aberration in the input image can be corrected in the subsequent process, and the accuracy of the obtained output image can be improved.

[0018] In a possible implementation, the aforementioned aberration estimation on the input image according to the hyperspectral image to obtain the aberration estimation image includes: taking the hyperspectral image as an input of a pre-trained generator to output the aberration estimation image, that is, performing the aberration estimation online by using the generator; or performing the aberration estimation on the input image according to the hyperspectral image by using an offline calibration algorithm, that is, performing the aberration estimation by using the offline algorithm.

[0019] In the embodiments of the present application, the accurate aberration estimation can be implemented in an offline or online manner, so as to facilitate subsequent correction of the aberration.

[0020] In a possible implementation, the aforementioned fusion of the first feature and the second feature to obtain the fused feature can further include: aligning the first feature and the second feature, and fusing the aligned first feature and the second feature to obtain the fused feature.

[0021] In the embodiments of the present application, the first feature and the second feature can be aligned in the spatial dimension, so as to improve the accuracy of the fusion of the first feature and the second feature, and obtain more accurate fused features.

[0022] In a second aspect, the present application provides an image processing device, comprising:

[0023] a first input module configured to acquire an input image, the input image being an image collected by an image sensor;

[0024] a second input module configured to acquire a hyperspectral image, the spectral channels of the hyperspectral image being different from the spectral channels of the input image;

[0025] a first feature extraction module configured to extract a feature from the input image to obtain a first feature;

[0026] a second feature extraction module configured to obtain a second feature according to a feature extracted from the hyperspectral image;

[0027] a feature fusion module configured to fuse the first feature and the second feature to obtain a fused feature;

[0028] a recovery module configured to obtain an output image according to the fused feature.

[0029] Effects of the second aspect and any optional implementation of the second aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, which will not be repeated here.

[0030] In a possible implementation, the feature fusion module is specifically configured to fuse the first feature and the second feature based on an attention mechanism to obtain the fused feature.

[0031] In a possible implementation, the feature fusion module is further configured to: acquire a historical frame image, the historical frame image being a previous frame image of the input image; extract a third feature from the historical frame image; and fuse the first feature, the second feature, and the third feature to obtain the fused feature.

[0032] In a possible implementation, the second feature extraction module is specifically configured to: filter information of a spectral channel from the hyperspectral image to obtain a spectral channel reconstruction image; and obtain the second feature according to a feature extracted from the spectral channel reconstruction image.

[0033] In a possible implementation, the second feature extraction module is specifically configured to: extract a third feature from the hyperspectral image; obtain a dynamic attention coefficient and a spectral feature according to the third feature based on an attention mechanism; and fuse the dynamic attention coefficient, the spectral feature, and a preset static attention coefficient to obtain the spectral channel reconstruction image.

[0034] In a possible implementation, the second feature extraction module is specifically configured to: perform at least one down-sampling on the spectral channel reconstruction image to obtain at least one down-sampled spectral image; and extract at least one scale feature from the at least one down-sampled spectral image, the second feature including the at least one scale feature.

[0035] In a possible implementation, the feature fusion module is further configured to: perform at least one down-sampling on the first feature to obtain at least one down-sampled feature, a scale of the at least one down-sampled feature corresponding to a scale of the at least one scale feature in a one-to-one manner; fuse the at least one down-sampled feature and the corresponding scale feature to obtain at least one scale fused feature; and fuse the at least one scale fused feature to obtain the fused feature.

[0036] In a possible implementation, the image processing apparatus further includes:

[0037] The aberration estimation module is configured to perform aberration estimation according to the hyperspectral image to obtain an aberration estimation image, the aberration estimation image being used to represent errors of pixel points in the input image.

[0038] The restoration module is specifically configured to perform restoration on the fused feature according to the aberration estimation image to obtain an output image.

[0039] In a possible implementation, the aberration estimation module is specifically configured to: take the hyperspectral image as an input of a pre-trained generator to output the aberration estimation image; or perform aberration estimation on the input image according to the hyperspectral image by using an offline calibration algorithm to obtain the aberration estimation image, the offline calibration algorithm being an algorithm determined according to preset steps in an offline state.

[0040] In a possible implementation, the feature fusion module is further configured to align the first feature and the second feature, and fuse the aligned first feature and the second feature to obtain a fused feature.

[0041] In a third aspect, an embodiment of the present application provides an image processing apparatus, including a processor and a memory, wherein the processor and the memory are interconnected through a circuit, the processor invokes program codes in the memory to execute functions related to processing in the image processing method shown in any of the first aspect or the optional implementation of the first aspect. Optionally, the electronic device can be a chip.

[0042] In a fourth aspect, an embodiment of the present application provides an electronic device, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is configured to execute functions related to processing in the first aspect or any of the optional implementations of the first aspect.

[0043] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, including instructions, when running on a computer, causing the computer to execute the method in the first aspect or any of the optional implementations of the first aspect.

[0044] In a sixth aspect, an embodiment of the present application provides a computer program product including instructions, when running on a computer, causing the computer to execute the method in the first aspect or any of the optional implementations of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0045] FIG. 1 is a structural schematic diagram of an electronic device provided by the present application;

[0046] FIG. 2 is a schematic diagram of a system architecture provided by the present application;

[0047] FIG. 3 is a schematic diagram of the method provided by the present application compared with the prior art;

[0048] FIG. 4 is a flowchart of an image processing method provided by the present application;

[0049] FIG. 5 is a structural schematic diagram of an image processing model provided by the present application;

[0050] FIG. 6 is a structural schematic diagram of another image processing model provided by the present application;

[0051] FIG. 7 is a structural schematic diagram of another image processing model provided by the present application;

[0052] FIG. 8 is a structural schematic diagram of another image processing model provided by the present application;

[0053] FIG. 9 is a structural schematic diagram of another image processing model provided by the present application;

[0054] FIG. 10 is a structural schematic diagram of another image processing model provided by the present application;

[0055] FIG. 11 is a structural schematic diagram of another image processing model provided by the present application;

[0056] FIG. 12 is a structural schematic diagram of another image processing model provided by the present application;

[0057] FIG. 13 is a structural schematic diagram of another image processing model provided by the present application;

[0058] FIG. 14 is a structural schematic diagram of another image processing model provided by the present application;

[0059] FIG. 15 is a structural schematic diagram of another image processing model provided by the present application;

[0060] FIG. 16 is a structural schematic diagram of another image processing model provided by the present application;

[0061] FIG. 17 is a structural schematic diagram of another image processing model provided by the present application;

[0062] FIG. 18 is a structural schematic diagram of an image processing device provided by the present application;

[0063] FIG. 19 is a structural schematic diagram of another image processing device provided by the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0065] First, the embodiments of the present application relate to a large number of neural network related applications. In order to better understand the solutions of the embodiments of the present application, the related terms and concepts of neural networks that may be involved in the embodiments of the present application will be introduced below.

[0066] (1) Neural network

[0067] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with x s (i.e. input data) and intercept 1 as input. The output of the operation unit can be:

[0068] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weight parameters are denoted by b, which represents the bias of the neural unit. f is the activation function of the neural unit, used to introduce non-linear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0069] (2) Loss Function

[0070] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually a pre-configuration process before the first update, where parameters are pre-configured for each layer.) For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the deep neural network becomes a process of minimizing this loss. Common loss functions include mean squared error, cross-entropy, logarithmic, and exponential loss functions. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0071] (3) Backpropagation algorithm

[0072] An algorithm for calculating the gradient of model parameters according to a loss function, updating the model parameters. A neural network can use a back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forward propagation of the input signal until the output produces an error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0073] In the training phase of the model, the BP algorithm can be used to train the model to obtain the trained model.

[0074] (4) Attention (also known as attention mechanism)

[0075] The attention mechanism can quickly extract important features of sparse data. The attention mechanism occurs between the encoder and the decoder, or between the input sentence and the generated sentence. The self-attention mechanism in the self-attention model occurs inside the encoding matrix or the output sequence, and can extract the relationship between words far apart in the same sentence, such as syntactic features (phrase structure). The self-attention mechanism provides an effective modeling method for capturing global context information through QKV. Assuming that the input is Q (query), and the context is stored in the form of key-value pairs (K, V). Then the attention mechanism is actually a mapping function from query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query to a series of (key, value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in the form of (K, V), then attention completes the addressing by calculating the similarity between Q and K. The similarity calculated by Q and K reflects the importance of the extracted V value, that is, the weight, and then the weighted sum is obtained. The final feature value.

[0076] The calculation of attention mainly includes three steps. The first step is to calculate the similarity between the query and each key to obtain the weight. Common similarity functions include dot product, concatenation, and perception. The second step is to normalize the weights using a softmax function, which can normalize the sum of all weight coefficients to 1 and highlight the weight of important elements. The final step is to weight and sum the corresponding key values to obtain the final feature value. The specific calculation formula can be as follows:

[0077] where d is the dimension of the matrix QK.

[0078] In addition, attention includes self-attention and cross-attention. Self-attention can be understood as a special attention, that is, the input of QKV is consistent. The input of QKV in cross-attention is inconsistent. Attention uses the similarity between features (such as inner product) as weight to integrate the queried features as the updated value of the current feature. Self-attention is an attention extracted based on the attention of the feature map itself.

[0079] For convolution, the size of the receptive field is limited by the setting of the convolution kernel, which often requires multiple layers of stacking to focus on the entire feature map. The advantage of self-attention is that its attention is global, and it can obtain the global spatial information of the feature map through simple query and assignment. The special point of self-attention in the query, key, and value (QKV) model is that the input corresponding to QKV is consistent.

[0080] (5) Raw data: Raw data records the original information of the camera sensor and is in an unprocessed and uncompressed format. RAW can be conceptualized as "original image encoding data" or more figuratively as "digital negative".

[0081] (6) R (red), G (green), B (blue)

[0082] where R represents red, G represents green, and B represents blue. Each image can be represented by the color values of these three channels. For example, an RGB image represents an image with three color channels, and an RGGB image represents an image with four color channels, two of which are G.

[0083] (7) Hyperspectral image

[0084] The hyperspectral sensor is used to image in the ultraviolet, visible, near-infrared or mid-infrared region of the electromagnetic spectrum. The number of channels of the hyperspectral image is usually determined by the number of wave bands of the spectrum collected by the hyperspectral sensor.

[0085] The method provided in the present application can be applied to various image processing scenarios, such as image denoising, image enhancement, image super-resolution, etc.

[0086] Specifically, in terms of the deployment manner of the method provided in the present application, the method provided in the present application can be divided into various deployment manners. For example, the method provided in the present application can be deployed in an electronic device, and a user can directly use the electronic device to perform image processing, or the method provided in the present application can be deployed in a cloud platform to provide image processing services for a user terminal. The different deployment manners will be introduced respectively as follows.

[0087] Deployment manner one, deployed in an electronic device

[0088] The electronic device provided in the embodiments of the present application can specifically include a handheld device, a vehicle-mounted device, a data processing apparatus, a computing device, etc., which includes or is connected with an image sensor. It can also include a digital camera, a cellular phone, a camera, a smart phone, a personal digital assistant (PDA) computer, a tablet computer, a laptop computer, a machine type communication (MTC) terminal, a point of sales (POS), a vehicle-mounted computer, a head-mounted device, a data processing apparatus (such as a bracelet, a smart watch, etc.), a security device, a virtual reality (VR) device, an augmented reality (AR) device, and other electronic devices with imaging functions.

[0089] Taking a digital camera as an example, a digital camera is a digital camera, which is a kind of camera that converts optical image into digital signal by using photoelectric sensor. Unlike traditional cameras that rely on the change of photosensitive chemical substances on film to record images, the sensor of a digital camera is a light-sensitive charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS). Compared with traditional cameras, digital cameras have the advantages of convenience, speed, repeatability, timeliness, etc. due to the direct use of photoelectric conversion image sensor. With the development of CMOS processing technology, the function of digital camera is becoming more and more powerful, and it has almost completely replaced traditional film cameras, and has extremely wide application in consumer electronics, human-computer interaction, computer vision, automatic driving and other fields.

[0090] Exemplarily, FIG. 1 shows a schematic diagram of an electronic device provided by the present application. As shown in the figure, the electronic device can include a lens group 110, an image sensor 120 and an electrical signal processor 130. The electrical signal processor 130 can include an analog-to-digital (A / D) converter 131 and a digital signal processor 132. The analog-to-digital converter 131 is an analog signal to digital signal converter, which is used to convert analog electrical signal to digital electrical signal.

[0091] It should be understood that the electronic device shown in FIG. 1 is not limited to including the above devices, and can also include more or less other devices, such as batteries, flashlights, keys, sensors, etc. The embodiments of the present application only take an electronic device with an image sensor 120 as an example for description, but the elements installed on the electronic device are not limited thereto.

[0092] In the embodiments of the present application, the aforementioned sensor 120 can specifically include an image sensor, a multispectral image (MSI) sensor, etc.

[0093] The light signal reflected by the photographed object is converged by the lens group 110 and imaged on the sensor 120. The image sensor 120 converts the light signal into an analog electrical signal. The analog electrical signal is converted into a digital electrical signal by the analog-to-digital (A / D) converter 131 in the electrical signal processor 130, and the digital electrical signal is processed by the digital signal processor 132, for example, the data electrical signal is optimized by a series of complex mathematical algorithm operations, and finally the image is output. The electrical signal processor 130 can also include an analog signal preprocessor 133 for pre-processing the analog electrical signal transmitted by the image sensor and outputting it to the analog-to-digital converter 131.

[0094] The performance of the image sensor affects the quality of the final output image. The image sensor, also known as a photosensitive chip, a photosensitive element, etc., contains hundreds of thousands to millions of photoelectric conversion elements that generate electric charges when exposed to light and are converted into digital signals by an analog-to-digital converter chip. The image sensor includes a plurality of photosensitive elements that achieve imaging through photoelectric response.

[0095] The MSI sensor can simultaneously collect image signals of multiple spectral bands, and the multiple spectral bands include visible spectral bands and invisible spectral bands (such as infrared bands and ultraviolet bands), which have the potential to improve the noise reduction effect of RGB images and videos. In some scenarios (such as extremely dark light scenes), RGB images cannot obtain the details and texture information of the shooting scene, while MSI images can cover a wider spectral range, and the invisible spectral band can collect more detailed information. Therefore, the method provided in the present application can be specifically deployed in the electrical signal processor 130 of the electronic device, such as the digital signal processor 132, or other processors of the electronic device. The method provided in the present application can use hyperspectral images to denoise visible light images, that is, use the richer information included in the hyperspectral images to filter the noise of the visible light images to obtain denoised visible light images.

[0096] At the same time, the MSI image can cover more spectral bands in the visible spectral band range, and under the same exposure condition, the MSI sensor receives more photons, so that the obtained image has a higher signal-to-noise ratio (SNR). The MSI image with a higher SNR can be used as an additional input to assist and guide the RGB image denoising process, which can effectively improve the denoising effect and restore more texture details.

[0097] Deployment mode two, deployed in a cloud platform

[0098] The present application also provides a cloud platform, and one or more terminals access the platform. The method provided in the present application can be deployed in the cloud platform to provide image denoising or enhancement services for the terminal.

[0099] For example, FIG. 2 is an application scenario of the method provided in the present application, which can include a cloud platform 11 and a terminal 12. The cloud platform 11 and the terminal 12 can be connected through wired or wireless connection.

[0100] The cloud platform 11 can specifically include a server cluster with storage and processing functions. The method provided in the present application can be deployed in the cloud platform 11, which can specifically receive multiple frames of images from the terminal 12 and perform image processing based on the multiple frames of images, such as image denoising, enhancement, etc., and feed back the processed images to the terminal 12.

[0101] The terminal 12 can implement image processing by interacting with the cloud platform 11. The terminal can specifically include, but is not limited to, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a notebook computer, a smart car, and the like. The terminal 12 can transmit an image to the cloud platform 11, which can be an image captured by the terminal itself, an image including user input, or an image stored locally in the terminal, and the like. For example, the cloud platform can provide services for users through a client deployed in the terminal or a webpage in the terminal, and taking the deployment of the client in the terminal as an example, the user can send the image collected by the terminal to the cloud platform through the client deployed in the terminal, such as transmitting an RGB image and an MSI image, and the cloud platform 11 outputs a high-definition RGB image and feeds back the terminal 12 by denoising or enhancing the RGB image through the method provided in the present application.

[0102] In a possible scenario, it can also be applied to a scenario of multiple terminals, for example, the user can use other terminals different from the terminal 12 to capture the aforementioned RGB image and MSI image, and transmit the RGB image and MSI image to the aforementioned terminal 12, and the terminal 12 can feed back the enhanced image to the terminal 12 after uploading the RGB image and MSI image to the cloud platform 11 for image denoising or enhancement processing, and the terminal 12 can feed back the enhanced image to the terminal that captures the aforementioned RGB image and MSI image.

[0103] Generally, the imaging process of a digital image sensor is affected by noise, such as noise from an imaging module and a circuit, and the like. The noise is usually manifested as noise points, blurring, or artifacts in the image, which will cause the quality of the image to decrease. For example, in some scenarios, such as a dark light scenario or a strong light scenario, there can be a case of loss of details or texture of the image.

[0104] For example, in an existing scheme, RGB-assisted MSI texture detail enhancement can be used, the input is a low-quality MSI image and a high-quality RGB image, and the output is a high-quality MSI image. This scheme mainly uses the characteristic that the RGB image and video have higher spatial resolution to improve the enhancement and recovery effect of the MSI image. However, in this scheme, a denoised RGB image cannot be obtained.

[0105] For example, in an existing scheme, NIR / UV assisted RGB texture detail enhancement is used. In this type of scheme, the input is a single non-visible band image video and an RGB image video, and the output is a high-quality RGB image video. This scheme processes the NIR image and obtains the fusion weight of the RGB image, fuses the RGB image according to the fusion weight, and finally improves the quality of the output RGB image. However, this scheme can only be applied to specific scenarios and has requirements for the degradation mode of the input image, such as extremely dark light imaging, haze scene imaging, etc. In addition, this type of scheme only introduces non-visible bands and does not use the full band information of MSI, so the improvement effect is limited.

[0106] For example, in an existing scheme, it is found that the green channel data has higher signal-to-noise ratio and texture detail information, the input is a low-quality RGB image and a green channel image, and the output is a high-quality RGB image. Therefore, this scheme uses the green spectral channel data to extract the green channel guide feature, and fuses the green channel guide feature with the features in the RGB image restoration network to obtain the final output result. However, only a small amount of spectral information is used as guide information, and the effect is limited. And the spectral information and the RGB image are spatially aligned by default, so it cannot be applied to a multi-camera imaging system.

[0107] Therefore, the image processing method provided in the present application can utilize the characteristics of multispectral data to assist image restoration and improve image quality.

[0108] As shown in FIG. 3, the first type of scheme is to use RGB to assist MSI texture detail enhancement, and the second and third types of schemes are to use RGB to assist MSI texture detail enhancement and use green spectral channel data to enhance RGB images. Obviously, compared with the first type of scheme, the method provided in the present application uses RGB images and hyperspectral images as input and outputs high-quality RGB images. The input and output of the first type of scheme, the second type of scheme and the third type of scheme are all different.

[0109] The method flow provided in the present application will be introduced below.

[0110] Referring to FIG. 4, the flowchart of the image processing method provided in the embodiment of the present application is shown.

[0111] 401, acquire an input image.

[0112] The input image can specifically include an image captured by the electronic device described above, or an image read from storage, etc.

[0113] For example, if the method provided by the embodiment of the present application is deployed in an electronic device, the input image can be an image captured by the electronic device. The electronic device can refer to the description corresponding to the foregoing FIG. 1, which will not be described here again.

[0114] For another example, if the method provided by the embodiment of the present application is deployed in a cloud platform, the input image can be specifically an image received from a terminal. For example, a user can input an image in the terminal, or capture an image by using the terminal, and send the image to the cloud platform through the terminal.

[0115] Specifically, the input image can include an image of at least one channel. For example, the input image can include a raw image or an RGB image, and the like, so as to use a plurality of visible light denoising scenes.

[0116] Optionally, in the process of obtaining the input image, after obtaining the image collected by the image sensor, a preprocessing operation such as image alignment or color space conversion can be performed, so as to obtain an image that conforms to a preset specification, thereby facilitating further processing of the input image.

[0117] 402. Obtain a hyperspectral image.

[0118] The hyperspectral image (HSI) can be specifically an image obtained by using data collected by a hyperspectral sensor. The hyperspectral image can include image information of a plurality of channels.

[0119] The spectral channels corresponding to the hyperspectral image are not completely the same as the channels corresponding to the input image. For example, the number of channels of the hyperspectral image can be greater than the number of channels of the input image, or the hyperspectral image includes information of channels not included in the input image.

[0120] In addition, the collection time of the input image is the same as or less than a preset value from the collection time of the hyperspectral image, so as to ensure that the environment information included in the hyperspectral image is the same as or close to the environment information included in the input image, thereby facilitating subsequent recovery of the texture and details of the input image by using the hyperspectral image.

[0121] Optionally, in the process of obtaining the HSI, after obtaining the image collected by the HIS sensor, a preprocessing operation such as image alignment or color space conversion can be performed, so as to obtain an image that conforms to a preset specification, thereby facilitating further processing of the HIS.

[0122] It should be noted that the present application does not limit the execution order of steps 401 and 402. Step 401 can be executed first, step 402 can be executed first, or steps 401 and 402 can be executed simultaneously. The specific execution order can be determined according to actual application scenarios.

[0123] 403、extracting a feature from the input image to obtain a first feature.

[0124] After obtaining the input image, a feature can be extracted from the input image by using the neural network. For the sake of distinction, the feature extracted from the input image is referred to as the first feature.

[0125] In a possible implementation, in the subsequent fusion step, if the features of multiple scales need to be fused, multiple scale features can be extracted from the first feature when the first feature is extracted; or the input image can be down-sampled one or more times when the first feature is extracted, so as to obtain images of different scales, and features of different scales are extracted from the images of different scales, and the features of different scales are the first features.

[0126] 404、obtaining a second feature according to the feature extracted from the hyperspectral image.

[0127] After obtaining the hyperspectral image, a feature can be extracted from the hyperspectral image by using the neural network. For the sake of distinction, the feature extracted from the hyperspectral image is referred to as the second feature.

[0128] In a possible implementation, information of a required spectral channel can be screened from the hyperspectral image to obtain a spectral channel reconstruction image; and then the second feature is obtained according to the feature extracted from the spectral channel reconstruction image. Therefore, in the implementation of the present application, the information of the required spectral channel can be screened from the hyperspectral image, and the spectral channel reconstruction image is obtained by reconstruction, so that the feature of the required spectral channel is extracted.

[0129] Further, in a possible implementation, a third feature can be extracted from the hyperspectral image; a dynamic attention coefficient and a spectral feature are obtained based on the attention mechanism according to the third feature; and the spectral channel reconstruction image is obtained by fusing the dynamic attention coefficient, the spectral feature, and a preset static attention coefficient. Therefore, in the implementation of the present application, the required spectral channel can be reconstructed based on the attention mechanism, so that the feature of the required spectral channel is more accurately obtained.

[0130] Further, when the second feature is extracted, features of multiple scales can be extracted to assist in denoising of the input image from multiple scales. Specifically, the spectral channel reconstruction image can be down-sampled at least once to obtain at least one down-sampled spectral image; and features are extracted from the at least one down-sampled spectral image respectively to obtain at least one scale feature, and the second feature includes the at least one scale feature.

[0131] 405、fusing the first feature and the second feature to obtain a fused feature.

[0132] After the first feature and the second feature are extracted, the first feature and the second feature can be fused to obtain a fused feature. Specifically, the first feature and the second feature can be fused by weighted fusion or other fusion mechanisms, and the fused feature in which the first feature and the second feature are fused is output. The fused feature can be understood as a fusion of the features extracted from the input image and the hyperspectral features extracted from the hyperspectral image, that is, it carries the data distribution features of the input image and the hyperspectral features of the hyperspectral image. Therefore, the features extracted from the input image can be supplemented or improved in the texture or detail dimensions by using the hyperspectral features in the hyperspectral image, so as to obtain a fused feature containing richer features.

[0133] Optionally, the first feature is down-sampled at least once to obtain at least one down-sampled feature, and the scale of the at least one down-sampled feature corresponds to the scale of the at least one scale feature one by one; the at least one down-sampled feature and the corresponding scale feature are fused respectively to obtain at least one scale fused feature; and the at least one scale fused feature is fused to obtain the fused feature. In the embodiments of the present application, the features can be fused from multiple different scales, so that the features at different scales are fused, and therefore the features extracted from the visible light image can be smoothed by using the spectral features at different scales extracted from the hyperspectral image to achieve the denoising effect at different scales.

[0134] 406. An output image is obtained according to the fused feature.

[0135] After the fused feature is obtained, the image restoration can be performed by using the fused feature to obtain the final output image, that is, the output image after the input image is denoised.

[0136] In the embodiments of the present application, the features can be extracted from the input image to be denoised and the hyperspectral image respectively, the hyperspectral features representing richer details and textures can be extracted from the hyperspectral image, and the features extracted from the input image are fused to obtain the fused features, so as to achieve the denoising effect of the input image. Therefore, in the embodiments of the present application, the features extracted from the input image can be fused by using the features extracted from the hyperspectral image with less noise, so as to smooth the noise of the features of the input image. The fused features with less noise are obtained, so as to obtain the output image with less noise after the image restoration, and the denoising effect of the input image is achieved.

[0137] In a possible embodiment, before the first feature and the second feature are fused, the first feature and the second feature are aligned, and then the aligned first feature and the second feature are fused to obtain the fused feature.

[0138] In an optional embodiment, in order to correct the aberration of the input image, the aberration estimation can be performed based on the hyperspectral image, so as to correct the aberration of the input image by using the aberration estimation map. That is, the image restoration can be performed based on the aberration estimation map and the fused features, so that the aberration of each pixel in the image can be corrected by using the aberration estimation map, and a more accurate output image can be obtained.

[0139] Optionally, the hyperspectral image can be taken as the input of the pre-trained generator, and the output is the aberration estimation map; or, the input image is aberration-estimated according to the hyperspectral image by using an offline calibration algorithm, and the aberration estimation map is obtained. In the embodiments of the present application, the aberration estimation of the input image can be realized in various ways, and more accurate aberration estimation can be realized.

[0140] The foregoing has introduced the method process provided by the present application in combination. The method process provided by the present application will be introduced in combination with a specific image processing model.

[0141] Exemplarily, the present application provides an image and video restoration system of a multi-modal camera, which is obtained from HSI sensors and RGB sensors. The RGB sensor can also be replaced by other visible light sensors, and the HSI sensor can also be replaced by other sensors for collecting multi-spectral images. The specific determination can be made according to the actual application scene. The RGB low-quality image or video is obtained by the RGB camera, and the HSI image or video is used to guide the realization of the quality restoration functions such as denoising and detail restoration of the RGB image or video.

[0142] Generally, for single-frame images and videos, different modules can be set in combination with the method process corresponding to the foregoing FIG. 4.

[0143] For example, for single-frame image processing, referring to FIG. 5, the present application provides a structural schematic diagram of an image processing model.

[0144] The image processing model can specifically include but is not limited to: an HSI preprocessing module 100, a spectral band screening and fusion module 201, a multi-scale feature extraction module 202, a multi-scale attention fusion module 300, an aberration estimation module 400, an RGB preprocessing module 500, and a restoration backbone network 600.

[0145] Among them:

[0146] The HSI preprocessing module 100 is used to receive the input HIS image: C in >3, C in The number of spectral channels of HIS is represented; image alignment, color space conversion and other processing operations are performed, and the processed For monocular imaging system, the image does not need to be spatially aligned and color space converted, so this module is not needed or is not enabled.

[0147] Band selection and fusion module 201: input is HSI image, output is feature tensor data. Specifically, it can receive input image or image from the output of the HIS preprocessing module 100: The band selection and fusion module contains a multi-layer convolution structure, which is used for feature extraction and spectral selection for spectral channel reconstruction, output That is, the aforementioned spectral channel reconstruction image.

[0148] Multi-scale feature extraction module 202: input filtered HSI image And small-scale HSI image after downsampling operation The spectral multi-scale guided feature extraction module contains a multi-layer convolution structure, which performs deep feature extraction on multi-scale images to improve the spectral feature representation capability and obtains multi-scale feature tensors That is, the aforementioned second feature, as the input of the subsequent fusion module.

[0149] Multi-scale attention fusion module 300: multi-scale attention-based feature space fusion module, input is the feature tensor obtained by the multi-scale feature extraction module 202 and the RAW / RGB image, output is the fused multiple feature tensors. Specifically, the input is the multi-scale feature of module 202 And the image of module 500 This module is composed of multi-layer convolution, transformer module, fully connected layer, upsampling and downsampling layer. The multi-layer convolution layer is used to extract the features in the image, that is, the aforementioned first feature, which is then passed through the transformer module, fully connected layer, upsampling and downsampling layer, etc. to realize the alignment and fusion of image features and spectral features, and obtain the fused features with consistent space and higher signal-to-noise ratio That is, the aforementioned fusion feature, for subsequent module processing.

[0150] Aberration estimation module 400: input is HSI image, output is optical aberration estimation map. Receive input HIS image: The optical aberration estimation module is composed of a point spread function generator, which estimates the optical aberration online, or directly uses the offline calibration method to obtain the optical aberration estimation map

[0151] RGB preprocessing module 500: receives input RGB image: C v ​∈ [1, 3], image alignment, color space conversion and other processing modules, output the processed Monocular system, no need for spatial alignment and color space conversion of the image, then no need for this module.

[0152] Denoising and detail restoration backbone network 600: input is the feature Z of module 300 and the image Denoising and detail restoration backbone network (also known as restoration backbone network) contains multiple layers of convolution structure for feature extraction and image reconstruction, output C out ∈ [1, 3], that is, the output image of the foregoing.

[0153] For example, for video image processing, referring to FIG. 6, the present application provides an architecture diagram of an image processing model.

[0154] The difference from the foregoing FIG. 5 is that for RGB image input, a processing channel for historical frames is also set. As shown in FIG. 6, when denoising the Tthframe RGB image, the T-1thframe RGB image is also introduced, and features are extracted from the Tthframe RGB image and the T-1thframe RGB image, as shown in the feature extraction module 601 in FIG. 6, so that in the multi-scale attention fusion module 300, the features extracted from the Tthframe RGB image and the T-1thframe RGB image can be fused to combine the information in the historical frames to further assist the current frame in denoising and improve the image denoising effect.

[0155] Among them, the summary of the steps performed by each module can be combined with the method flow of the foregoing FIG. 4, and the specific functions and structures of each main module are introduced below.

[0156] 1, spectral band selection and fusion module 201

[0157] Among them, the steps specifically performed by the spectral band selection and fusion module 201 can be referred to the description of the step 404 in the foregoing FIG. 4 about reconstructing the spectral channel to output the reconstructed spectral channel image.

[0158] Exemplarily, the structure of the spectral band selection and fusion module 201 can be as shown in FIG. 7, which can include a convolution layer and one or more spectral channel attention blocks (SCAB). The convolution layer can be used to extract features, and the SCAB can be used to further extract the required spectral features based on the attention mechanism. The input of the spectral band selection and fusion module 201 is The output is

[0159] Specifically, the structure of the SCAB can be as shown in FIG. 8, which includes a convolutional layer and a multi-layer perceptron (MLP) layer. The input of the SCAB can be represented as a feature tensor After passing through a convolutional layer and an MLP layer, a dynamic attention coefficient W1 (dimension 1x1xC f1 , i.e., an attention coefficient matrix dynamically calculated during image processing, and after being fused with the input feature, a dynamically fused spectral feature x f2 (i,j) = x f1 (i,j) x W1

[0160] The dynamically fused feature obtains a fused spectral feature, which, after passing through an activation function (here, the activation function is a Gaussian error linear unit (GELU)), is fused with a static attention parameter W2 (dimension C f2 xC f1 obtained during the training phase, to obtain a static fused spectral feature x i+1 (i,j) = x f2 (i,j) x W2

[0161] Thus, the fused spectral feature is output.

[0162] In the embodiments of the present application, features can be extracted from the input image based on the attention mechanism, and features of the required spectrum can be extracted based on dynamic attention and static attention, respectively, so as to screen out spectral features for subsequent auxiliary RGB image denoising.

[0163] 2. The multi-scale feature extraction module 202

[0164] The steps specifically performed by the multi-scale feature extraction module 202 can refer to the description of the steps of extracting features from the hyperspectral image in the foregoing step 404.

[0165] Exemplarily, the structure of the multi-scale feature extraction module 202 can be as shown in FIG. 9. The multi-scale feature extraction module 202 can include multiple scale processing channels, each channel can include a downsampling operation and a feature extraction module. The downsampling operation can be used to downsample the input image, and the feature extraction module can be used to extract features.

[0166] The input of the multi-scale feature extraction module 202 can be represented as: Through the feature extraction module, the input feature x In other channels, Through downsampling, a low-resolution input feature x Through the feature extraction module, the input feature x

[0167] In the embodiments of the present application, features of different scales can be extracted, so that when subsequent feature fusion is performed, the noise of the RGB image can be reduced from different scales, and the noise in the RGB image can be further smoothed by HIS features of different scales to improve the SNR of the output image.

[0168] 3. Multi-scale attention fusion module 300

[0169] The steps performed by the multi-scale attention fusion module 300 can be referred to the description in the foregoing step 405. Similarities will not be repeated here.

[0170] Specifically, the multi-scale attention fusion module 300 can be used to realize simultaneous fusion of spatial, temporal and spectral fusion. For example, the structure of the multi-scale attention fusion module 300 can be as shown in FIG. 10. In the embodiments of the present application, feature extraction and fusion can be performed from multiple scales, and a down-sampling module, a convolution layer and an up-sampling module can be respectively arranged. The down-sampling module can be used to down-sample the input data, the convolution layer can be used to extract features, and a spatial time and spectral fusion module (Spectral-Spatial-Temporal Transformer Block, STB). For the input RGB image (i.e. the current frame shown in FIG. 10) Historical frame Features can be extracted from different scales respectively, and the same scale spectral features extracted by the STB are fused respectively, so that the RGB features (i.e. the features extracted from the RGB image) and the spectral features are fused from different scales, and the low-scale features after fusion are up-sampled to restore the scale.

[0171] Specifically, the network structure of the STB is different for different scenes of images and videos.

[0172] For example, for image restoration scenes, taking one scale as an example, the structure of the STB as a feature spectral spatial fusion module can be as shown in FIG. 11. The STB divides the features in the space into non-overlapping feature blocks, and performs fusion operation on the feature blocks at the same position. For example, for different scale features Linear projection is performed by the attention mechanism, which is divided into And the input image Linear projection is performed based on the attention mechanism, which is divided into For And And And The inner product calculation is performed respectively, and the calculated inner products are normalized respectively and spliced, and the spliced features are multiplied by and to output the fused features.

[0173] For example, for a video recovery scene, the STP is extended to a feature spectrum, space, and time fusion module, and the structure of the STB can be as shown in FIG. 12. The fusion operation is performed on different time feature blocks at the same position. The difference from the foregoing FIG. 11 is that a fusion process of scale features extracted from historical frames is added, that is, the is linearly projected and divided into The inner product of and and is calculated, and in the multiplication process, the inner product of and and is added, so as to obtain the fused features fused with the historical frame features.

[0174] 4. Aberration estimation module 400

[0175] The aberration estimation module 400 can be used to implement the step of performing aberration estimation to output an aberration estimation map described in the foregoing step 406.

[0176] Specifically, the aberration estimation method can include various methods, such as online distortion and point spread function estimation, or offline distortion and point spread function estimation, which will be introduced below.

[0177] (1) Online distortion and point spread function estimation

[0178] For example, as shown in FIG. 13, HSI can be used as the input of the point spread function generator, and the optical aberration estimation map is output. The point spread function generator is pre-trained. The structure of the point spread function generator can be as shown in FIG. 14, and the point spread function generator can include multiple convolution layers.

[0179] (2) Offline distortion and point spread function estimation

[0180] The optical aberration estimation map is obtained by using an offline calibration method, and the offline calibrated optical aberration estimation map is directly used as the input of the denoising and detail recovery network in the network inference stage. The offline calibration flowchart is shown in FIG. 15. When the image is collected, subsequent spectral alignment, spatial alignment, and blur kernel estimation are performed, and In the online inference stage, each step of offline calibration can be directly used to output the corresponding aberration estimation map.

[0181] 5. The restored backbone network 600

[0182] The restored backbone network 600 can be used to perform the step of obtaining the output image in the aforementioned step 406. Specifically, the pre-trained restored backbone network can be used to output the final denoised output image.

[0183] For example, as shown in FIG. 16, the fused features and the aberration estimation map can be input into the pre-trained restored backbone network, and the image restoration step can be performed by the restored backbone network to obtain the output image.

[0184] In the embodiments of the present application, an image processing architecture is provided, which can be applied to various image restoration tasks with RAW / RGB images as input, such as denoising, demosaicing, and super-resolution, etc. The modules in the architecture provided by the present application can also be migrated to other types of image restoration and enhancement tasks that require RAW / RGB images. In addition, the image processing scheme provided by the present application has high robustness to real open scenes, and can adapt to RAW images collected by different sensor models and different imaging scenes.

[0185] Correspondingly, the loss value to be calculated in the training stage can be as shown in FIG. 17. The loss value L1 loss between the aberration estimation map and the true aberration map and the loss value L1 loss between the output RGB image and the true image can be calculated, and the model is updated in reverse based on the loss value, so as to obtain the updated image processing model. Subsequently, the image processing model can be deployed to denoise the input image based on HIS using the deployed image processing model in the inference stage. That is, in the training stage, the L1 reconstruction error (L1 loss) and other general reconstruction losses are used to constrain the network output and the true image (Ground Truth); at the same time, the L1 reconstruction error can be used to constrain the optical aberration estimation map in the optical aberration estimation module 400 and the auxiliary true image.

[0186] The L1 loss can be expressed as:

[0187] Therefore, in the inference stage, the trained model can be directly used to perform the inference task.

[0188] Therefore, in the method provided by the embodiments of the present application, an RGB image / video denoising and detail restoration framework based on hyperspectral guidance is implemented, T v (T v ≥1) frame RGB image and T h (T h ≥1) frame full-spectrum hyperspectral image are input, and T o (T o≥1) Frame denoising and detail enhancement of RGB image. Light guides the feature extraction of spectral image, which is used for subsequent RGB fusion and denoising, so as to fuse the RGB feature with the spectral feature, increase the texture and details of the RGB feature with the spectral feature, and reduce the noise in the RGB feature. And the spectral feature-RGB feature fusion and denoising non-independent algorithm module is directly completed by a parameter-optimizable network, and the denoising result is output.

[0189] Compared with the existing scheme, such as compared with only using a single or a small number of wave bands, the incremental information is small, and the effect is limited. The method provided in the application can utilize full-spectrum information, can provide more texture information and higher signal-to-noise ratio auxiliary data, and significantly improve the RAW or RGB denoising and detail recovery effect.

[0190] For example, compared with the fusion of HSI and RGB in the spectral dimension, the spatial registration performance of the HSI and RGB binocular system is relatively high. The scheme provided in the application simultaneously performs fusion processing on the HSI image and the RGB image in the spectral, spatial and temporal dimensions. The feature fusion module is uniformly optimized with other modules in the embodiment of the application in an end-to-end manner, so as to improve the fusion effect and the final RGB image / video recovery effect.

[0191] In addition, in the scheme provided in the application, optical aberration estimation and correction are provided, and blur and chromatic aberration caused by optical distortion are processed and recovered. Specifically, the aberration of the imaging system can be corrected in an online or offline manner, and more details can be recovered.

[0192] The foregoing describes the method provided in the application, and the structure of the device for executing the foregoing method provided in the application is described below.

[0193] Referring to FIG. 18, the application provides a structural schematic diagram of an image processing device, which comprises:

[0194] The first input module 1801 is configured to acquire an input image, and the input image is an image collected by an image sensor;

[0195] The second input module 1802 is configured to acquire a hyperspectral image, and the spectral channels of the hyperspectral image are not completely the same as the spectral channels of the input image;

[0196] The first feature extraction module 1803 is configured to extract a feature from the input image to obtain a first feature;

[0197] The second feature extraction module 1804 is configured to obtain a second feature according to the feature extracted from the hyperspectral image;

[0198] The feature fusion module 1805 is configured to fuse the first feature and the second feature to obtain a fused feature;

[0199] The restoration module 1806 is configured to obtain an output image according to the fused feature.

[0200] In a possible implementation, the feature fusion module 1805 is specifically configured to fuse the first feature and the second feature based on an attention mechanism to obtain the fused feature.

[0201] In a possible implementation, the feature fusion module 1805 is further configured to: obtain a historical frame image, the historical frame image being a previous frame image of the input image; extract a third feature from the historical frame image; and fuse the first feature, the second feature, and the third feature to obtain the fused feature.

[0202] In a possible implementation, the second feature extraction module 1804 is specifically configured to: filter information of a spectral channel from the hyperspectral image to obtain a spectral channel reconstruction image; and obtain the second feature according to a feature extracted from the spectral channel reconstruction image.

[0203] In a possible implementation, the second feature extraction module 1804 is specifically configured to: extract a third feature from the hyperspectral image; obtain a dynamic attention coefficient and a spectral feature based on the third feature according to an attention mechanism; and fuse the dynamic attention coefficient, the spectral feature, and a preset static attention coefficient to obtain the spectral channel reconstruction image.

[0204] In a possible implementation, the second feature extraction module 1804 is specifically configured to: perform at least one down-sampling on the spectral channel reconstruction image to obtain at least one down-sampled spectral image; and extract a feature from each of the at least one down-sampled spectral image to obtain at least one scale feature, the second feature including the at least one scale feature.

[0205] In a possible implementation, the feature fusion module 1805 is further configured to: perform at least one down-sampling on the first feature to obtain at least one down-sampled feature, a scale of the at least one down-sampled feature corresponding to a scale of the at least one scale feature in a one-to-one manner; fuse each of the at least one down-sampled feature and the corresponding scale feature to obtain at least one scale fused feature; and fuse the at least one scale fused feature to obtain the fused feature.

[0206] In a possible implementation, the image processing apparatus further includes:

[0207] The aberration estimation module 1807 is configured to perform aberration estimation according to the hyperspectral image to obtain an aberration estimation image, the aberration estimation image being used to represent errors of pixel points in the input image.

[0208] The restoration module 1806 is specifically configured to restore the fused feature according to the aberration estimation image to obtain an output image.

[0209] In a possible implementation, the aberration estimation module 1807 is specifically configured to: input the hyperspectral image into a pre-trained generator to output an aberration estimation image; or perform aberration estimation on the input image according to the hyperspectral image to obtain the aberration estimation image, by using an offline calibration algorithm.

[0210] In a possible implementation, the feature fusion module 1805 is further configured to: align the first feature and the second feature, and fuse the aligned first feature and second feature to obtain the fused feature.

[0211] As shown in FIG. 19, it is a schematic diagram of a hardware structure of an image processing apparatus 190 provided by an embodiment of the present application. The image processing apparatus 190 can be used to implement the steps of the methods in the foregoing FIGS. 4-17.

[0212] The image processing apparatus 190 shown in FIG. 19 can include a processor 1901, a memory 1902, a communication interface 1903, and a bus 1904. The processor 1901, the memory 1902, and the communication interface 1903 can be connected through the bus 1904.

[0213] The processor 1901 is the control center of the image processing apparatus 190, which can be a general central processing unit (CPU), or other general processors, etc. The general processor can be a microprocessor or any conventional processor, etc., which can specifically include a GPU or an NPU, etc., and can be adaptively set according to actual application scenarios.

[0214] As an example, the processor 1901 can include one or more CPUs, and can also include other processors, such as the CPU, NPU, or GPU shown in FIG. 19, etc.

[0215] The memory 1902 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or can be an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0216] In a possible implementation, the memory 1902 can exist independently of the processor 1901. The memory 1902 can be connected to the processor 1901 through the bus 1904, for storing data, instructions or program codes. When the processor 1901 invokes and executes the instructions or program codes stored in the memory 1902, the method provided by the embodiments of the present application can be implemented, for example, the method shown in FIGS. 4 to 17.

[0217] In another possible implementation, the memory 1902 can also be integrated with the processor 1901.

[0218] The communication interface 1903 is configured to connect the image processing apparatus 190 to other devices through a communication network, which can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), or the like. The communication interface 1903 can include a receiving unit for receiving data, and a sending unit for sending data.

[0219] The bus 1904 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in FIG. 19, but it does not mean that there is only one bus or only one type of bus.

[0220] It should be noted that the structure shown in FIG. 19 does not constitute a limitation on the image processing apparatus 190. In addition to the components shown in FIG. 19, the image processing apparatus 190 can include more or fewer components than shown, or combine certain components, or different arrangement of components.

[0221] Those skilled in the art can clearly understand the application by the description of the foregoing embodiments that the application can be realized by means of software necessary general hardware, of course, can also be realized by special hardware including special integrated circuit, special CPU, special memory, special components, etc. Generally, the functions completed by the computer program can be easily realized by the corresponding hardware, and the specific hardware structure for realizing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, the software program implementation is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of software product in essence or in the form of software product that makes contribution to the prior art, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., including a plurality of instructions for making a device (which can be a personal computer, server, or network equipment, etc.) execute the method described in various embodiments of the application.

[0222] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product.

[0223] The computer readable storage medium in the embodiments of the application stores a program for training a model or performing an inference task, and when the program runs on a computer, the computer executes all or part of the steps of the method described in the foregoing embodiments of FIG. 3 to FIG. 8.

[0224] The embodiments of the application also provide a digital processing chip. The digital processing chip integrates a circuit for realizing the processor or the function of the processor and one or more interfaces. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, the digital processing chip can be connected with an external memory through a communication interface. The digital processing chip realizes the method steps of any one or more of the foregoing embodiments according to the program code stored in the external memory.

[0225] The embodiments of the present application further provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0226] The data comparison device provided by the embodiments of the present application can be a chip, which includes a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin, a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the server executes the method described in the embodiments shown in FIGS. 4 to 17. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0227] In particular, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0228] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0229] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components, etc. Generally, any function completed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special-purpose circuits, etc. However, for the present application, software program implementation is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0230] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product.

[0231] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0232] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or inverse order, depending upon the circumstances. The term "and / or" in the present application is merely used to represent an association between associated objects, and it is possible that three relationships exist, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or modules as an example is not necessarily limited to those steps or modules that are clearly listed, but can include other steps or modules that are not clearly listed or inherent to such process, method, product or device. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering, and the named or numbered flow steps can be executed in a different order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in the present application is a logical division, and in actual application, there can be another division manner, for example, multiple modules can be combined or integrated in another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some ports, and the indirect coupling or communication connection between the modules can be electrical or other similar forms, which are not limited in the present application. Furthermore, the modules or sub-modules described as separate components can or can not be physically separated, and can or can not be physical modules, or can be distributed to multiple circuit modules, and part or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present application.

Claims

1. An image processing method, characterized by, The method comprises the following steps: obtaining an input image, wherein the input image is an image collected by an image sensor; obtaining a hyperspectral image, wherein a spectral channel of the hyperspectral image is not completely the same as a spectral channel of the input image; extracting a feature from the input image to obtain a first feature; extracting a feature from the hyperspectral image to obtain a second feature; fusing the first feature and the second feature to obtain a fused feature; obtaining an output image according to the fused feature.

2. The method of claim 1, wherein, The step of fusing the first feature and the second feature to obtain a fused feature comprises the following steps: fusing the first feature and the second feature based on an attention mechanism to obtain the fused feature.

3. The method according to claim 1 or 2, characterized in that, The step of fusing the first feature and the second feature to obtain a fused feature further comprises the following steps: obtaining a historical frame image, wherein the historical frame image is a previous frame image of the input image; extracting a third feature from the historical frame image; fusing the first feature, the second feature and the third feature to obtain the fused feature.

4. The method according to any one of claims 1-3, characterized in that, The step of extracting a feature from the hyperspectral image to obtain a second feature comprises the following steps: screening information of a spectral channel from the hyperspectral image to obtain a spectral channel reconstruction image; extracting a feature from the spectral channel reconstruction image to obtain the second feature.

5. The method of claim 4, wherein, The step of screening information of a preset spectral channel from the hyperspectral image to obtain a spectral channel reconstruction image comprises the following steps: extracting a third feature from the hyperspectral image; obtaining a dynamic attention coefficient and a spectral feature based on the third feature according to an attention mechanism; fusing the dynamic attention coefficient, the spectral feature and a preset static attention coefficient to obtain the spectral channel reconstruction image.

6. The method according to claim 4 or 5, characterized in that, The step of extracting a feature from the spectral channel reconstruction image to obtain the second feature comprises the following steps: performing at least one down-sampling on the spectral channel reconstruction image to obtain at least one down-sampled spectral image; extracting at least one scale feature from the at least one down-sampled spectral image respectively, wherein the second feature comprises the at least one scale feature.

7. The method of claim 6, wherein, The step of fusing the first feature and the second feature to obtain a fused feature further comprises the following steps: performing at least one down-sampling on the first feature to obtain at least one down-sampled feature, wherein a scale of the at least one down-sampled feature corresponds to a scale of the at least one scale feature respectively; fusing the at least one down-sampled feature and the corresponding scale feature respectively to obtain at least one scale fused feature; fusing the at least one scale fused feature to obtain the fused feature.

8. The method according to any one of claims 1-7, characterized in that, The method further comprises the following steps: performing aberration estimation on the hyperspectral image to obtain an aberration estimation image, wherein the aberration estimation image is used to represent errors of pixel points in the input image; The step of obtaining an output image according to the fused feature comprises the following step: recovering the fused feature according to the aberration estimation image to obtain the output image.

9. The method of claim 8, wherein, The step of performing aberration estimation on the hyperspectral image to obtain an aberration estimation image comprises the following step: inputting the hyperspectral image into a pre-trained generator to output the aberration estimation image. Or, aberration estimation is performed on the input image according to the hyperspectral image by using an offline calibration algorithm, and an aberration estimation image is obtained, wherein the offline calibration algorithm is an algorithm determined according to preset steps in an offline state.

10. The method according to any one of claims 1-9, characterized in that, The fusing the first feature and the second feature to obtain a fusion feature further includes: The first feature and the second feature are aligned, and the aligned first feature and the second feature are fused to obtain the fusion feature.

11. An image processing apparatus characterized by comprising: It includes: The first input module is configured to obtain an input image, wherein the input image is an image collected by an image sensor. The second input module is configured to obtain a hyperspectral image, wherein a spectral channel of the hyperspectral image is not completely identical to a spectral channel of the input image. The first feature extraction module is configured to extract a feature from the input image to obtain a first feature. The second feature extraction module is configured to obtain a second feature according to a feature extracted from the hyperspectral image. The feature fusion module is configured to fuse the first feature and the second feature to obtain a fusion feature. The restoration module is configured to obtain an output image according to the fusion feature.

12. The apparatus of claim 11, wherein The feature fusion module is specifically configured to fuse the first feature and the second feature based on an attention mechanism to obtain the fusion feature.

13. The apparatus of claim 11 or 12, wherein, The feature fusion module is further configured to: Obtain a historical frame image, wherein the historical frame image is a previous frame image of the input image. Extract a third feature from the historical frame. Fuse the first feature, the second feature, and the third feature to obtain the fusion feature.

14. The apparatus of any one of claims 11-13, wherein, The second feature extraction module is specifically configured to: Filter information of a spectral channel from the hyperspectral image to obtain a spectral channel reconstruction image. Obtain the second feature according to a feature extracted from the spectral channel reconstruction image.

15. The apparatus of claim 14, wherein, The second feature extraction module is specifically configured to: Extract a third feature from the hyperspectral image. Obtain a dynamic attention coefficient and a spectral feature based on the third feature based on an attention mechanism. Fuse the dynamic attention coefficient, the spectral feature, and a preset static attention coefficient to obtain the spectral channel reconstruction image.

16. The apparatus of claim 14 or 15, wherein, The second feature extraction module is specifically configured to: At least once down-sample the spectral channel reconstruction image to obtain at least one down-sampled spectral image. Extract a feature from each of the at least one down-sampled spectral image to obtain at least one scale feature, and the second feature includes the at least one scale feature.

17. The apparatus of claim 16, wherein, The feature fusion module is further configured to: At least once down-sample the first feature to obtain at least one down-sampled feature, wherein a scale of the at least one down-sampled feature corresponds to a scale of the at least one scale feature one by one. Fuse the at least one down-sampled feature and the corresponding scale feature to obtain at least one scale fusion feature. Fuse the at least one scale fusion feature to obtain the fusion feature.

18. The apparatus of any one of claims 11-17, wherein, The apparatus further includes: an aberration estimation module configured to perform aberration estimation on the hyperspectral image to obtain an aberration estimation map, the aberration estimation map being used to represent errors of pixel points in the input image; the restoration module is specifically configured to restore the fused feature according to the aberration estimation map to obtain the output image.

19. The apparatus of claim 18, wherein, The aberration estimation module is specifically configured to: output the aberration estimation map by taking the hyperspectral image as an input of a pre-trained generator; or, perform aberration estimation on the input image according to the hyperspectral image by using an offline calibration algorithm to obtain an aberration estimation map, the offline calibration algorithm being an algorithm determined according to pre-set steps in an offline state.

20. The apparatus of any one of claims 11-19, wherein, The feature fusion module is further configured to: align the first feature and the second feature, and fuse the aligned first feature and second feature to obtain the fused feature.

21. An image processing apparatus characterized by comprising: One or more processors, the one or more processors and the memory are coupled, the memory stores a program, when the program stored by the memory is executed by the one or more processors, the steps of the method in any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium, characterized in that, The program is executed by the processing unit, and the method in any one of claims 1 to 10 is executed.

23. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps of the method in any one of claims 1 to 10.

Images