Imaging sensor, image processing device, and image processing method

Abstract

An image processing apparatus has circuitry configured to obtain global image data comprising image data and spectral data, the global image data representing a global image region, and the circuitry is configured to input the global image data to a machine learning model for generating output spectral data, wherein the neural network is configured to transform the obtained image data to the output spectral data based on the obtained spectral data.

Description

Technical Field

[0001] This disclosure generally relates to imaging sensors, image processing apparatus, and image processing methods. Background Technology

[0002] Neural networks, such as deep neural networks (DNNs) and convolutional neural networks (CNNs), are generally known and are used in various technical fields, such as image processing. Known image processing devices can use DNNs and CNNs for image reconstruction, multi-spatial and multispectral image transformation, multi-spatial and multispectral image generation, object recognition, and more.

[0003] Furthermore, DNNs and CNNs typically have an input layer, an output layer, and multiple hidden layers between the input and output layers. In image processing, neural networks can be trained to output images with high spectral resolution, using color channel images captured by color sensors, such as RGB images (with red, green, and blue color channels), and spectral channel images captured by multispectral sensors, such as multispectral or hyperspectral images, as input to the neural network.

[0004] Although technologies for image processing already exist, there is a general desire to improve imaging sensors, image processing devices, and methods. Summary of the Invention

[0005] According to a first aspect, this disclosure provides an image processing apparatus, including circuitry configured to acquire global image data comprising image data and spectral data representing a global image region, and to input the global image data into a machine learning model to generate output spectral data, wherein the machine learning model is configured to transform the acquired image data into output spectral data based on the acquired spectral data.

[0006] According to a second aspect, this disclosure provides an image processing method, including acquiring global image data including image data and spectral data, the global image data representing a global image region, and inputting the global image data into a machine learning model to generate output spectral data, wherein the machine learning model is configured to transform the acquired image data into output spectral data based on the acquired spectral data.

[0007] Further aspects are set forth in the dependent claims, the following description, and the accompanying drawings. Attached Figure Description

[0008] The implementation method is described below with reference to the accompanying drawings, wherein:

[0009] Figure 1 An exemplary implementation of an imaging sensor is shown;

[0010] Figure 2 A block diagram illustrating an implementation of the image processing system is shown;

[0011] Figure 3 This illustrates an implementation of a global image region, in which spectral information is distributed;

[0012] Figure 4 This illustrates an implementation of a global image region, in which spectral information is clustered.

[0013] Figure 5 This illustrates an implementation of spectral information grouped within regions of a global image area;

[0014] Figure 6 Visualizing the applications of convolutional neural networks;

[0015] Figure 7 A block diagram illustrating an implementation scheme for the learning system is shown.

[0016] Figure 8 This illustrates an implementation scheme for the learning method of convolutional neural networks;

[0017] Figure 9 An implementation of the proposed system is shown, which is based on using Figure 8 The learning method is used to train the machine learning model; and

[0018] Figure 10 A flowchart illustrating an implementation of the image processing method is shown. Detailed Implementation

[0019] Provide a reference Figure 1 Before providing a detailed description of the implementation methods, a general description will be given.

[0020] As mentioned at the beginning, multispectral imaging systems and conventional red-green-blue (RGB) imaging systems are generally known for capturing and analyzing images with high spectral resolution and high spatial resolution, respectively. Typically, multispectral imaging devices provide higher resolution spectral information than conventional RGB imaging systems, which usually only provide color channel information—red, green, and blue. Multispectral sensing devices are generally more expensive than RGB imaging devices, and often involve a trade-off between resolution, acquisition time, and cost.

[0021] Generally speaking, the spatial resolution of a mosaic array multispectral sensor may be lower than that of a regular RGB sensor. However, since the design cost of a regular RGB sensor is usually lower than that of a multispectral sensor, most imaging systems focus on spatial resolution rather than spectral resolution.

[0022] As is well known, multispectral imaging systems use deep learning techniques to reconstruct hyperspectral / multispectral image data based on RGB images in order to benefit from spatial and spectral resolution information.

[0023] As mentioned above, neural networks, such as deep neural networks (DNNs) and convolutional neural networks (CNNs), are known and have achieved state-of-the-art performance in many fields, such as image processing, image reconstruction, multi-spatial and multispectral image generation, and language processing. CNNs are a type of DNN and are commonly used for analyzing visual images.

[0024] Specifically, CNNs use image classification algorithms for image transformation, multi-spatial and multispectral image generation, image classification, image and video recognition, natural language processing, and more.

[0025] As is well known, a CNN can have an input layer, an output layer, and multiple hidden layers. The hidden layers of a CNN typically consist of multiple convolutional layers, such as pooling layers and fully connected layers. Each convolutional layer in a neural network usually has attributes, such as an input with a shape (number of images) × (image width) × (image height) × (image depth), and many convolutional kernels. The convolutional kernels act like filters; their width and height are hyperparameters, and their depth must generally be equal to the depth of the image. A convolutional layer convolves the input and passes the result to the next layer.

[0026] In some cases, it may be appropriate to train a conventional CNN to reconstruct hyperspectral / multispectral images, for example, from RGB images captured by an RGB sensor and hyperspectral images captured by a hyperspectral sensor. In this case, the CNN can be trained to integrate data from different capture systems and perform alignment and / or interpolation processing on the RGB image data representing the captured RGB images to the hyperspectral / multispectral image data representing the captured hyperspectral / multispectral images.

[0027] However, it has been recognized that, for example, in certain scenarios, multispectral imaging is required, combining high resolution with large-scale acquisition (depending on acquisition time) or rapid acquisition (depending on acquisition time). In such cases, it has been recognized that conventional CNNs may not be suitable because the resolution of hyperspectral / multispectral images is higher than that of RGB images; that is, hyperspectral / multispectral images have hundreds of pixels more than RGB images, and information loss may occur due to the alignment and / or interpolation processing of the two different images. Furthermore, in such cases, two different sensors are required to capture two different images.

[0028] Therefore, some implementations involve imaging sensors that include a photosensitive region, wherein the photosensitive region includes an imaging unit and a spectral imaging unit.

[0029] The imaging sensor can be any high-resolution / high-speed RGB imaging sensor, etc. The imaging sensor can be specifically designed to sense light of different wavelength ranges based on filters applied to its pixels, where the pixels can be arranged in a pixel array in the sensor plane. At least one pixel of the sensor can be a multi / hyperspectral pixel with sensitivity in a multispectral or hyperspectral range. The image sensor can generate image data based on the wavelength range of the incident light.

[0030] In some implementations, the photosensitive region can be a region formed by a pixel array, such as a photosensitive sensor plane or a region corresponding to a photosensitive sensor plane. The photosensitive region includes an imaging unit and a spectral imaging unit. The photosensitive region can be based on at least one of the following: CMOS (Complementary Metal-Oxide-Semiconductor), CCD (Charge-Coupled Device), SPAD (Single-Photon Avalanche Diode), CAPD (Current-Assisted Photodiode) technology, etc.

[0031] The imaging unit can be a common color channel imaging unit, such as a common RGB (red, green, blue) imaging unit with multiple pixels, used to sense light of different wavelength ranges, i.e., red, green, and blue light, without limiting this disclosure in this respect. The multiple pixels of the imaging unit can sense any other light color (channel).

[0032] Specifically, a color channel for a particular color (e.g., red, green, or blue) can include information from multiple spectral channels corresponding to the wavelength ranges of red, green, or blue, respectively. In other words, a color channel can be considered as an integration of corresponding spectral channels(s) within the wavelength range of the relevant color channel.

[0033] The spectral imaging unit can be a spectral channel imaging unit, which has, for example, at least one imaging portion for sensing light of different wavelength ranges or spectral ranges. Specifically, the spectral imaging unit can be specifically designed to sense light of different wavelength ranges or spectral ranges based on filters applied to at least one imaging portion. For example, different types of color filters, spectral filters, etc., can be applied to the spectral imaging unit, different from the color filters, spectral filters, etc., applied to the RGB pixels of the imaging unit. Therefore, the spectral pixels can be sensitive to wavelength ranges or spectral ranges different from those of the RGB pixels in the photosensitive area of ​​the imaging sensor.

[0034] In some embodiments, the spectral imaging unit may include multiple spectral imaging portions, such as pixels. The multiple spectral imaging portions may be distributed within a photosensitive region, or at least a portion of the multiple spectral imaging portions may be grouped within a region of the photosensitive region, or at least a cluster comprising multiple spectral imaging portions may be distributed within the photosensitive region, or the spectral imaging portions may be located within the photosensitive region in any combination of all of the above. Each pixel may be configured to detect light within a specific spectral range of the entire multispectral or hyperspectral wavelength or spectral range to be detected.

[0035] Multiple spectral imaging portions can be arbitrarily distributed within the photosensitive region, or they can be distributed based on a predetermined pattern. Furthermore, multiple spectral imaging portions can be embedded together with the imaging unit (and / or the pixels of the imaging unit) within the photosensitive region.

[0036] For example, return to Figure 1 An exemplary embodiment of an imaging sensor, as discussed herein, is shown, which includes an imaging unit and a spectral imaging unit.

[0037] As mentioned above, imaging sensors, for example Figure 1 The imaging sensor 1 has a photosensitive region 2, which further includes an imaging unit 3 and a spectral imaging unit 4. The imaging unit 3 is represented by white colored squares, and the spectral imaging unit 4 is represented by checkerboard pattern squares. In this embodiment, the spectral imaging unit 4 has multiple spectral imaging portions distributed in the photosensitive region 2. The imaging unit 3 and the spectral imaging unit 4 may each have a filter for appropriately filtering the wavelength of the incident light. For example, information such as imaging information (e.g., red, green, and blue color channel information) is acquired from the imaging unit 3, and information such as spectral information is acquired from the spectral imaging unit 4. The information acquired from the imaging unit 3 and the spectral imaging unit 4 is represented by image data and spectral data, respectively.

[0038] Imaging sensor 1 generates, for example, global image data, including image data and spectral data representing imaging information and spectral information, respectively, which will be referred to below. Figures 3 to 5 Further description.

[0039] Some implementations relate to an image processing apparatus, including circuitry configured to acquire global image data comprising image data and spectral data representing a global image region, and to input the global image data into a machine learning model for generating output spectral data, wherein the machine learning model is configured to transform the acquired image data into output spectral data based on the acquired spectral data.

[0040] Image processing devices can be digital (video) cameras, surveillance cameras, biometric devices, security cameras, medical / health care devices, remote sensing devices, food inspection devices, image sensors that support edge computing (such as smart sensors associated with smart speakers), motor vehicle equipment, smartphones, personal computers, laptop computers, wearable electronic devices, electronic glasses, etc., circuits, processors, multiprocessors, logic circuits, or a combination of these components.

[0041] The circuit may include one or more processors, logic circuits, memory (read-only memory, random access memory, storage memory, i.e., hard disk, optical disk, flash drive, etc.), interface for communication via a network (e.g., wireless network, Internet, local area network, etc.), CMOS (complementary metal-oxide-semiconductor) image sensor, CCD (charge-coupled device) image sensor, etc., and may include the image sensor discussed herein.

[0042] As described above, global image data can be generated by an image sensor. Global image data can also be obtained from memory included in the device, from external memory, or from artificial image generators created using computer-generated graphics.

[0043] Image data can be red, green, and blue (RGB) image data. Image data can be represented by multiple color channels, such as three color channels, such as red, green, and blue. A color channel for a specific color (e.g., red, green, or blue) can include information from multiple spectral channels corresponding to the wavelength ranges of red, green, or blue, respectively. That is, a color channel can be considered as an integration of corresponding spectral channels(s) within the wavelength range of the relevant color channel. Furthermore, global image data represents a global image region, which is formed, for example, by multiple pixels.

[0044] Spectral data can be represented by any number of spectral channels. Spectral data can be represented by multiple spectral channels and can be, for example, multispectral or hyperspectral data.

[0045] The aforementioned image processing device can be part of an image processing system, such as a digital camera, RGB camera, surveillance camera, image acquisition system, professional camera, industrial equipment, etc. See below for reference. Figure 2 An exemplary embodiment of an image processing system including an image processing apparatus is described.

[0046] Figure 2 A block diagram illustrating an embodiment of an image processing system is shown, the image processing system including the image processing apparatus as described herein.

[0047] An image processing system (e.g., a digital camera 11) has an image sensor (e.g., as described in the above reference). Figure 1 The described components include an imaging sensor 1), an image processing device 12, and a display 13. In this embodiment, the image processing device 12 includes circuitry 17 with an interface 19; a central processing unit (CPU) 20 including multiple processors, including a graphics processing unit (GPU); a memory 21 including RAM, ROM, and storage memory; and a trained CNN 22 (stored in the memory).

[0048] Image processing device 12 acquires data, such as global image data 14, through interface 19. This data includes image data (e.g., image data 15) and spectral data (e.g., spectral data 16). Global image data 14 represents an image of the target scene captured by digital camera 11. In this embodiment, image data 15 is represented by multiple color channels, namely red, green, and blue. Furthermore, image data 15 represents data from... Figure 1 The imaging information corresponds to the information acquired by the imaging unit 3 of the photosensitive area 2 of the imaging sensor 1. The spectral data 16 is multispectral image data represented by multiple spectral channels. The spectral data 16 represents spectral information, which is related to the information acquired from the imaging sensor 1. Figure 1 The information obtained corresponds to the spectral imaging unit 4 of the photosensitive region 2 of the imaging sensor 1. In this embodiment, from Figure 1 The acquisition of spectral information by multiple spectral imaging portions grouped in the upper right region of the photosensitive region 2 of the imaging sensor 1 does not limit this aspect of the disclosure.

[0049] Global image data 14, represented by multiple color channels, is transmitted to CPU 20. CPU 20 inputs global image data 14 into CNN 22 to generate multispectral data represented by multiple spectral channels. CNN 22 has been pre-trained to transform acquired image data 15 into multispectral data based on acquired spectral data 16, for example, outputting spectral data 18. Output spectral data 18 represents the output image displayed on display 13.

[0050] The aforementioned image processing system 11, comprising an image sensor 1, can generate high-resolution multispectral imaging through full-resolution RGB (or similar) imaging and low-resolution multispectral imaging. Therefore, a multi / hyperspectral device with low cost, high resolution, and high acquisition speed can be designed.

[0051] CNN 22 can be a CNN trained from scratch using a machine learning algorithm, or it can be a CNN trained using a previously trained machine learning algorithm.

[0052] As described above, the global image data 14 includes image data 15 and spectral data 16, representing, for example, a global image region formed by multiple pixels. In some embodiments, the global image region may include more pixels represented by the acquired image data than pixels represented by the acquired spectral data (i.e., pixels representing spectral information may be sparser than pixels representing imaging information (RGB information)). The acquired spectral data may represent spectral information, and the acquired image data may represent imaging information. In some embodiments, the imaging information may correspond to the imaging portion of the photosensitive area of ​​the imaging sensor (e.g., ...). Figure 1 The information is obtained from the imaging unit 3 of the photosensitive region 2 of the imaging sensor 1. Therefore, the spectral information can correspond to the information obtained from the spectral imaging unit (e.g., the spectral imaging unit of the photosensitive region of the imaging sensor). Figure 1 The information acquired by the spectral imaging unit 4 of the photosensitive region 2 of the imaging sensor 1.

[0053] As described herein, exemplary embodiments of global image data include image data representing imaging information and spectral data representing spectral information, distributed across a global image region, as follows: Figures 3 to 5 As shown in the image.

[0054] Figure 3 An implementation representing a global image region of global image data is shown, wherein spectral information acquired from the spectral imaging portion of an imaging sensor is distributed. Specifically, as discussed herein, an imaging sensor (e.g., imaging sensor 1) generates global image data 14, which includes image data 15 and spectral data 16. Global image data 14 represents a global image region, such as image 31. Image data 15 represents imaging information 32, and spectral data 16 represents spectral imaging information 33. From the photosensitive region distributed in the imaging sensor (e.g., Figure 1 The imaging sensor 1 acquires spectral imaging information 33 from multiple spectral imaging portions in the photosensitive region 2). That is, sub-spectral imaging information 34 is acquired from each of the multiple spectral imaging portions. Therefore, in this embodiment, the spectral imaging information 33 is distributed throughout the global image region.

[0055] exist Figure 4 The diagram illustrates an implementation of spectral information acquired from the spectral imaging portion of an imaging sensor and clustered across the global image region. (See reference...) Figure 3The imaging sensor (e.g., imaging sensor 1) generates global image data 14, including image data 15 and spectral data 16. Global image data 14 represents a global image region, such as image 31, in which imaging information 32 and spectral imaging information 33 are distributed. Spectral imaging information 33 is obtained from multiple spectral imaging portions, which are clustered and distributed within the photosensitive area of ​​the imaging sensor, such as... Figure 1 The photosensitive area 2 of the imaging sensor 1. Therefore, in this embodiment, the spectral imaging information 33 is distributed in clusters 35 in the global image region.

[0056] Figure 5 Another implementation of spectral information is illustrated, which is acquired from the spectral imaging portion of an imaging sensor and grouped within regions of a global image area. As described herein, the imaging sensor (e.g., imaging sensor 1) generates global image data 14, including image data 15 and spectral data 16. The spectral information is grouped within the photosensitive region of the imaging sensor (e.g., Figure 1 The imaging sensor 1 acquires spectral imaging information 33 represented by spectral data in multiple spectral imaging portions within the photosensitive region 2). Therefore, in this embodiment, the spectral imaging information 33 is grouped in region 36 at the upper right corner of the global image region.

[0057] The above reference Figures 3 to 5 The description of the embodiments does not limit this disclosure in this respect. Spectral imaging information can be distributed in the global image region in any combination of the distribution methods described above.

[0058] In some implementations, the machine learning model may be a neural network, specifically a convolutional neural network (CNN), without limiting the scope of this disclosure. For example, in some implementations, the CNN may include convolutional layers, or it may also include local pooling layers or global pooling layers, such as max pooling layers, which reduce the dimensionality of the image data, as is commonly known. Pooling layers may be used for pooling, a form of non-linear downsampling, such as spatial pooling, i.e., max pooling, average pooling, sum pooling, etc.

[0059] The generation of output spectral data can occur during the training phase of a neural network (such as a CNN), or it can be generated using a pre-trained neural network (such as a trained CNN), for example, to extract information from image data (such as object recognition, or to identify other information in image data, such as spatial information, spectral information, patterns, colors, etc.). Therefore, the neural network can be a trained neural network or an untrained neural network, where an untrained neural network can be trained on the fly, for example, during the operation of the associated (image processing) device.

[0060] Furthermore, the neural network can be part of an image processing device, such as being stored in the memory or storage of the image processing device, or the image processing device can access the neural network, for example, via inter-processor communication, electronic bus, network (including the Internet).

[0061] Figure 6 The general principles of CNN usage are illustrated exemplarily. The first row of the diagram shows the general CNN structure, and the second row shows the basic principles of the building blocks. The principles of CNNs and their applications in imaging are generally known; therefore, the following explanation is for reference only. Figure 6 Brief discussion.

[0062] The input image consists of, for example, three maps or layers (typically red, green, and blue (RGB) color information) and N by N blocks. A CNN has convolutional layers and subsequent pooling layers, where this structure can be repeated, as shown below. Figure 6 As shown in the diagram. Convolutional layers consist of neurons. By applying kernels (filters) to the input image (see convolutional kernels in the second row), corresponding feature maps can be obtained. Pooling layers in this embodiment are based on max pooling (see "max pooling" in the second row), capturing information from the most active neurons in the convolutional layer and discarding other information. After several repetitions ( Figure 6 This process (which occurs three times in total) ends with a fully connected layer, also known as an affine layer. The final layer typically contains a number of neurons, corresponding to the number of object categories (output features) the CNN aims to distinguish. The output is... Figure 6 The first row is shown as the output distribution, which is represented by a row and columns, where each column represents a category and the height of the column represents the weight of that object category. Different categories correspond to the output or image attribute features of the CNN. These categories are, for example, "people, cars, etc." Hundreds or thousands of categories can typically be used, for example, for object recognition of different objects.

[0063] In some implementations, the convolutional neural network (CNN) can be trained based on acquired image data and acquired spectral data. Therefore, in some implementations, the CNN can be trained to transform acquired image data into output spectral data based on spectral information obtained from multiple spectral imaging portions of the photosensitive region of an imaging sensor.

[0064] As discussed in this article, a CNN can be either a trained CNN or an untrained CNN. Training options for CNNs, as discussed above, are detailed in the following reference. Figure 7 describe.

[0065] Figure 7An implementation of the learning system 40 is illustrated in block diagram form. This system generates machine learning models and trains neural networks such as CNN 22 on these models.

[0066] The learning system 40 includes data acquisition 41, dataset 42, training of a system (such as a machine learning algorithm) from scratch 43a, or training of a pre-trained system (such as a machine learning algorithm based on a pre-trained model 44) 43b, and machine learning models 45a and 45b.

[0067] Global image data 14 includes image data 15 and spectral data 16, representing a series of images, such as one hundred to one thousand (100-1000) sub-images acquired in data acquisition 41, and thus, a dataset 42 is generated and stored in the memory 21 of the image processing device 12. The number of one hundred to one thousand (100-1000) images is a relatively large number, resulting in sufficient data to train a machine learning algorithm. The machine learning algorithm can be trained using training 43a, for example, training from scratch. CNN 22 can be trained using a machine learning algorithm, for example, using spectral information 33 as the underlying facts, which is derived from data included in... Figure 1 The image information 32 is acquired by the spectral imaging section of the spectral imaging unit 4 of the imaging sensor 1, and has, for example, multiple multi / hyperspectral pixels. The CNN 22 is also trained based on spectral data 16 representing spectral information 33, transforming image information 32 into multi / hyperspectral information. Figure 1 The imaging sensor 1 acquires information in its imaging unit 3, which has, for example, red, green, and blue (RGB) pixels, and multi / hyperspectral information, for example, from... Figure 2 The information represented by the output spectral data 18. The learning system 40 generates a learning model 45a, which is stored in the memory 21 of the image processing device 12.

[0068] Alternatively, the training 43b of the pre-trained system can be used to perform the same transformation, requiring less data acquisition. The pre-trained system can be trained based on the pre-trained model 44. The learning system 40 generates a learning model 45b, which is stored in the memory 21 of the image processing device 12.

[0069] The training process can be implemented in the image processing system 11 (e.g., a camera, in a cloud service, in a user computer, in a dedicated device, etc.).

[0070] Some implementations relate to image processing methods that can be performed by the image processing apparatus described herein, or by any other electronic device, processor, or other computing device. The method includes acquiring global image data, which includes image data and spectral data representing a global image region, and inputting the global image data into a machine learning model to generate output spectral data, wherein the machine learning model is configured to transform the acquired image data into output spectral data based on the acquired spectral data.

[0071] As described above, compared to pixels represented by the acquired spectral data, the global image region can include more pixels represented by the acquired image data. Furthermore, the acquired spectral data can represent spectral information, and the acquired image data can represent imaging information. The spectral information can correspond to information acquired from the spectral imaging portion of the photosensitive region of the imaging sensor. Moreover, the machine learning model is a neural network; specifically, it can be a convolutional neural network, which can be trained based on the acquired image data and the acquired spectral data. Furthermore, the convolutional neural network can be trained to transform the acquired image data into output spectral data based on spectral information acquired from multiple spectral imaging portions of the photosensitive region of the imaging sensor. The convolutional neural network can be trained based on a learning algorithm that computes a loss function corresponding to the image data and a loss function corresponding to the spectral data.

[0072] Reference Figure 8 This illustrates an implementation of the processing scheme of the learning method 50 of CNN 22, wherein the acquired image data 15 is transformed into output spectral data 18 based on the acquired spectral data 16.

[0073] Image processing device 12 inputs (arrow 56) global image data 14 to CNN 22. Global image data 14 includes image data 15, such as RGB image data, and spectral data 16, such as multispectral / hyperspectral data. As discussed herein, global image data 14 represents a global image region, where the number of pixels represented by image data 15 is greater than the number of pixels represented by spectral data 16. Therefore, the RGB image represented by global image data 14 has sparse multispectral / hyperspectral information. CNN 22 predicts spectral data (arrow 57), such as multispectral / hyperspectral data, and outputs it as output spectral data 18 (e.g., after subsequent learning processes). The predicted spectral data is inversely transformed (arrow 58) into image data 52 using inverse transform 51. Inverse transform 51 is performed using a physical model and, in particular, by integrating the spectrum according to specific color sensitivities (red, green, blue). Based on image data 15 and the inversely transformed image data 52, an RGB loss function 53 indicating the loss in the RGB domain is calculated and then fed into CNN 22. Accordingly, based on the spectral data 16 and the predicted spectral data, a spectral loss function 54 indicating the loss in the spectral domain is computed and then fed into the CNN 22. For example, the spectral loss function 54 can be a one-to-one comparison between the predicted spectral data and the closest available (e.g., based on RGB values) multispectral / hyperspectral data in the sparse spectral information. Furthermore, smoothing 55, such as minimizing the total variation, is performed on the predicted spectral data. The RGB loss function 53 and the spectral loss function 54 can be used consecutively, for example, by first using the loss in the RGB domain, such as the RGB loss function 53, training the convolutional neural network under the smoothness constraint of smoothing 55, and then fine-tuning the learning models 45a, 45b using the sparse multispectral / hyperspectral information.

[0074] exist Figure 9 The proposed system 60 is illustrated in block diagram form, wherein the image processing system 11 is trained using learning method 50 based on machine learning models 45a and 45b, and the image processing system 11 includes an imaging sensor 1, an image processing device 12, and a display 13.

[0075] As described above, the imaging sensor 1 is a high-resolution / high-speed RGB sensor that acquires global image data 14, including image data 15 representing a high-resolution RGB image and spectral data 16 representing a low-resolution multi / hyperspectral image. The global image data 14 is input to the image processing device 12 to generate output spectral data 18 representing the multi / hyperspectral image, which can be displayed on the display 13. The output spectral data 18 is generated by a CNN 22, which is trained using a learning method 50 based on a machine learning model 45 (45a or 45b). As discussed herein, using the learning method 50, the image data 15 is transformed into the output spectral data 18 based on the spectral data 16.

[0076] Below, for reference Figure 10 The image processing method 70 performed by the image processing apparatus 12 and / or the image processing system 11 in some embodiments is discussed.

[0077] At 71, global image data, such as global image data 14 as described above, is acquired by the image processing device 12 and / or the image processing system 11.

[0078] Global image data can be obtained from an image sensor such as imaging sensor 1, or from a memory included in the device, from an external memory, or from an artificial image generator created by computer-generated graphics, etc.

[0079] At 72, global image data is fed into a convolutional neural network, such as CNN 22, to generate output multispectral (MS) data at 73, such as output spectral data 18 as described above.

[0080] Global image data includes image data and spectral data. Image data can be represented by multiple color channels such as red, green, and blue.

[0081] As mentioned above, global image data represents a global image region, which includes more pixels represented by the acquired image data compared to pixels represented by spectral data.

[0082] In 74, the convolutional neural network transforms the acquired image data into output spectral data based on the acquired spectral data.

[0083] Image data represents imaging information, and the acquired spectral data represents spectral information, which corresponds to the information acquired from the spectral imaging unit of the photosensitive region of the imaging sensor.

[0084] At position 75, output the resulting spectral image data.

[0085] It should be recognized that the embodiments describe an exemplary ordering of method steps. However, the specific ordering of the method steps is for illustrative purposes only and should not be construed as a constraint.

[0086] In some embodiments, the methods described herein are also implemented as computer programs that, when executed on a computer and / or processor, cause the computer and / or processor to perform the methods. In some embodiments, a non-transitory computer-readable recording medium is also provided, storing a computer program product that, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0087] All units and entities described in this specification and claimed in the appended claims, unless otherwise stated, can be implemented as integrated circuit logic, for example on a chip, and unless otherwise stated, the functions provided by these units and entities can be implemented by software.

[0088] With respect to the above-described embodiments of the present disclosure being implemented at least in part using a software-controlled data processing apparatus, it is understood that both the computer program providing such software control and the transmission, storage, or other medium providing such computer program are contemplated as aspects of the present disclosure.

[0089] Note that the technology disclosed herein can also be configured as described below.

[0090] (1) An imaging sensor, comprising:

[0091] The photosensitive region includes the imaging section and the spectral imaging section.

[0092] (2) The imaging sensor according to (1), wherein the spectral imaging unit includes multiple spectral imaging sections.

[0093] (3) The imaging sensor according to (2), wherein multiple spectral imaging portions are distributed in the photosensitive region.

[0094] (4) The imaging sensor according to (2), wherein at least a portion of the plurality of spectral imaging portions are grouped in a region of the photosensitive region.

[0095] (5) An image processing apparatus, comprising a circuit configured to:

[0096] Acquire global image data, which includes image data and spectral data, and the global image data represents the global image region;

[0097] Global image data is input into a neural network to generate output spectral data, wherein the neural network is configured to transform the acquired image data into output spectral data based on the acquired spectral data.

[0098] (6) The image processing apparatus according to (5), wherein the global image region includes more pixels represented by the acquired image data than the pixels represented by the acquired spectral data.

[0099] (7) The image processing apparatus according to (5) or (6), wherein the acquired spectral data represents spectral information and the acquired image data represents imaging information.

[0100] (8) The image processing apparatus according to (7), wherein the spectral information corresponds to the information obtained from the spectral imaging unit of the photosensitive region of the imaging sensor.

[0101] (9) The image processing apparatus according to any one of (5) to (8), wherein the neural network is a convolutional neural network.

[0102] (10) The image processing apparatus according to (9), wherein the convolutional neural network is trained based on the acquired image data and the acquired spectral data.

[0103] (11) The image processing apparatus according to (9), wherein the convolutional neural network is trained to transform the acquired image data into the output spectral data based on spectral information obtained from multiple spectral imaging portions of the photosensitive region of the imaging sensor.

[0104] (12) The image processing apparatus according to any one of (5) to (11), wherein the acquired image data is RGB image data and the acquired spectral data is multispectral data or hyperspectral data.

[0105] (13) An image processing method, comprising:

[0106] Acquire global image data, which includes image data and spectral data, and the global image data represents the global image region;

[0107] Global image data is input into a neural network to generate output spectral data, wherein the neural network is configured to transform the acquired image data into output spectral data based on the acquired spectral data.

[0108] (14) The image processing method as in (13), wherein the global image region comprises more pixels represented by the acquired image data than the pixels represented by the acquired spectral data.

[0109] (15) The image processing method according to (13) or (14), wherein the acquired spectral data represents spectral information and the acquired image data represents imaging information.

[0110] (16) According to the image processing method described in (15), wherein the spectral information corresponds to the information obtained from the spectral imaging unit of the photosensitive region of the imaging sensor.

[0111] (17) The image processing method according to any one of (5) to (16), wherein the neural network is a convolutional neural network.

[0112] (18) According to the image processing method described in (17), wherein a convolutional neural network is trained based on the acquired image data and the acquired spectral data.

[0113] (19) According to the image processing method described in (17), wherein the convolutional neural network is trained to transform the acquired image data into the output spectral data based on spectral information obtained from multiple spectral imaging units of the photosensitive region of the imaging sensor.

[0114] (20) According to the image processing method described in (17), a convolutional neural network is trained based on a learning algorithm, which calculates a loss function corresponding to the image data and a loss function corresponding to the spectral data.

[0115] (21) A computer program comprising program code that, when executed on a computer, causes the computer to perform the method according to any one of (13) to (20).

[0116] (22) A non-transitory computer-readable recording medium storing a computer program product that, when executed by a processor, causes the method according to any one of (13) to (20) to be performed.

Claims

1. An image processing apparatus, comprising circuitry configured to: Acquire global image data, which includes image data and spectral data, wherein the global image data represents a global image region. The acquired spectral data represents spectral information, which corresponds to information acquired from the spectral imaging unit of the photosensitive region of the imaging sensor, and wherein the image data represents imaging information acquired from the imaging unit of the photosensitive region; The global image data is input into a convolutional neural network to generate output spectral data. The convolutional neural network is configured to transform the acquired image data into the output spectral data based on the acquired spectral data. The global image region includes more pixels represented by the acquired image data than the number of pixels represented by the acquired spectral data. The convolutional neural network is trained using the following methods: The inverse transform uses a physical model that integrates the spectrum based on color sensitivity to convert the predicted spectral data into inversely transformed image data. A first loss function is calculated between the acquired image data and the inversely transformed image data; A second loss function is calculated between the acquired spectral data and the predicted spectral data; and Smoothing constraints are applied to the predicted spectral data.

2. The image processing apparatus according to claim 1, wherein, The acquired image data is RGB image data, and the acquired spectral data is multispectral data or hyperspectral data.

3. An image processing method, comprising: Acquire global image data, which includes image data and spectral data. The global image data represents a global image region. The acquired spectral data represents spectral information, which corresponds to information acquired from the spectral imaging unit of the photosensitive region of the imaging sensor. The image data represents imaging information acquired from the imaging unit of the photosensitive region. The global image data is input into a convolutional neural network to generate output spectral data. The convolutional neural network is configured to transform the acquired image data into the output spectral data based on the acquired spectral data. The global image region includes more pixels represented by the acquired image data than the number of pixels represented by the acquired spectral data. The convolutional neural network is trained using the following methods: The inverse transform uses a physical model that integrates the spectrum based on color sensitivity to convert the predicted spectral data into inversely transformed image data. A first loss function is calculated between the acquired image data and the inversely transformed image data; A second loss function is calculated between the acquired spectral data and the predicted spectral data; and Smoothing constraints are applied to the predicted spectral data.

Images