Image processing apparatus and image processing method

Abstract

An image processing apparatus has circuitry configured to obtain input image data represented by a plurality of color channels and to input the input image data to a neural network for generating output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein a number of spectral channels of the second multispectral image data is greater than a number of spectral channels of the first multispectral image data.

Description

Technical Field

[0001] This disclosure generally relates to an image processing apparatus and an image processing method. Background Technology

[0002] Neural networks such as deep neural networks (DNNs) and convolutional neural networks (CNNs) are generally known and used in various technical fields, such as image processing. Known image processing devices can use DNNs and CNNs for image reconstruction, multi-spatial and multispectral image generation, object recognition, and more.

[0003] Furthermore, DNNs and CNNs typically have an input layer, an output layer, and multiple hidden layers between the input and output layers. In image processing, neural networks can be trained to output images with high spectral or high spatial resolution, using color channel images as input to the neural network, such as RGB images (with red, green, and blue channels).

[0004] Despite the existence of techniques for image processing, there is a general desire to improve image processing devices and methods. Summary of the Invention

[0005] According to a first aspect, this disclosure provides an image processing apparatus, including a circuit configured to: acquire input image data represented by a plurality of color channels; input the input image data into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data.

[0006] According to a second aspect, this disclosure provides an image processing method, comprising: acquiring input image data represented by a plurality of color channels; inputting the input image data into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein the number of spectral channels of the second multispectral image data is greater than the number of spectral channels of the first multispectral image data.

[0007] Further aspects are set forth in the dependent claims, the following description, and the accompanying drawings. Attached Figure Description

[0008] The embodiments are explained by way of example with reference to the accompanying drawings, wherein:

[0009] Figure 1 A proposed method for generating multispectral image data from input image data represented by multiple color channels is shown;

[0010] Figure 2 An exemplary optimization relationship between spectral resolution and spatial resolution of multispectral image data is shown;

[0011] Figure 3 This demonstrates the application of convolutional neural networks;

[0012] Figure 4 A block diagram of an embodiment of an image processing apparatus is shown;

[0013] Figure 5 An embodiment of the processing scheme for the learning method of convolutional neural networks is shown;

[0014] Figure 6 A block diagram of an embodiment of the learning system is shown;

[0015] Figure 7 A block diagram illustrating an embodiment of the image processing system is shown; and

[0016] Figure 8 This is a flowchart of an embodiment of an image processing method. Detailed Implementation

[0017] Provide a reference Figure 4 Before a detailed description of the embodiments, a general explanation will be given.

[0018] As mentioned at the beginning, it is well known that multispectral imaging systems and conventional red-green-blue (RGB) imaging systems are used to capture and analyze images with high spectral resolution and high spatial resolution, respectively. Typically, multispectral imaging devices provide higher resolution spectral information than conventional RGB imaging systems. Analysis of high-resolution spectra can be used in a variety of applications, such as biostatistics, remote sensing, medicine, and food inspection. Multispectral sensing devices are generally more expensive than RGB imaging devices.

[0019] Furthermore, the spatial resolution of mosaic array multispectral sensors is typically lower than that of ordinary RGB sensors. However, since the design cost of ordinary RGB sensors is generally lower than that of multispectral sensors, most imaging systems focus on spatial resolution rather than spectral resolution.

[0020] As is well known, multispectral imaging systems use deep learning techniques to perform hyper / multispectral image data reconstruction from RGB images in order to benefit from spatial and spectral resolution information.

[0021] As mentioned at the beginning, neural networks such as deep neural networks (DNNs) and convolutional neural networks (CNNs) are known and have achieved state-of-the-art performance in many fields such as image processing, image reconstruction, multi-spatial and multispectral image generation, and language processing. CNNs are a type of DNN and are commonly used for analyzing visual images.

[0022] In particular, CNNs use image classification algorithms for image transformation, multi-spatial and multispectral image generation, image classification, medical image analysis, image and video recognition, natural language processing, and material classification applications (e.g., remote sensing, medical diagnosis).

[0023] As is well known, a CNN can have an input layer, an output layer, and multiple hidden layers. The hidden layers of a CNN typically consist of multiple convolutional layers, such as pooling layers and fully connected layers. Each convolutional layer in a neural network usually has attributes, such as an input of shape (number of images) x (image width) x (image height) x (image depth), multiple convolutional kernels acting as filters, whose width and height are hyperparameters, and whose depth must generally be equal to the depth of the image. The convolutional layer convolves the input and passes the result to the next layer.

[0024] In some cases, it may be appropriate to train a traditional CNN to reconstruct hyper / multispectral images from RGB images. In such cases, a traditional CNN can be trained to output only images with a predetermined number of spectral channels, without considering the amount of spectral information that might be needed depending on the application, target scene, system, user expectations, etc. Furthermore, this approach typically requires high computational power and a large amount of memory when computing high-resolution multispectral images with a large number of spectral channels. Additionally, traditional methods often output hyper / multispectral images with a predetermined number of spectral channels, thus potentially containing unnecessary target spectral information, and vice versa.

[0025] However, it has been recognized that, for example, different systems, applications, and target scenarios may require different spatial and spectral resolutions in the output image data. Furthermore, different proportions of spatial and spectral resolution in the output image data may be suitable for different systems, applications, or target scenarios. In such cases, it has been recognized that traditional CNNs may be unsuitable because by pre-setting a predetermined number of spectral channels, the output image data may include an unnecessary amount of spectral or spatial information.

[0026] Therefore, some embodiments relate to an image processing apparatus, including circuitry configured to: acquire input image data represented by a plurality of color channels; input the input image data into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data.

[0027] Image processing devices can be digital (video) cameras, surveillance cameras, biometric devices, security cameras, medical / health care devices, remote sensing devices, food inspection devices, image sensors that support edge computing (e.g., smart sensors associated with smart speakers), motor vehicle devices, smartphones, personal computers, laptop computers, wearable electronic devices, electronic glasses, etc., circuits, processors, multiple processors, logic circuits, or a combination of these components.

[0028] The circuit may include one or more processors, logic circuits, memory (read-only memory, random access memory, etc., storage memory, i.e., hard disk, optical disk, flash drive, etc.), interfaces for communication via networks such as wireless networks, the Internet, local area networks, CMOS (complementary metal-oxide-semiconductor) image sensors, CCD (charge-coupled device) image sensors, etc.

[0029] As described above, the input image data can be generated by an image sensor. Alternatively, the input image data can be obtained from memory included in the device, from external memory, or from an artificial image generator that creates graphics via computer generation.

[0030] Input image data can be represented by multiple color channels, for example, three color channels, such as red, green, and blue. Input image data can also be represented, for example, by a small number of spectral channels. A color channel for a specific color (e.g., red, green, or blue) can include information from multiple spectral channels corresponding to the wavelength ranges of red, green, or blue, respectively. That is, a color channel can be considered as an integration of the corresponding spectral channels(s) within the wavelength range of the relevant color channel.

[0031] refer to Figure 1 This paper presents a proposed method for generating multispectral image data from input image data represented by multiple color channels.

[0032] As described above, the image processing apparatus acquires input image data representing, for example, an image captured by a digital camera, as input image data 1. Input image data 1 is represented by multiple color channels. In this embodiment, the number of channels for input image data 1 is three, namely red, green, and blue, but this disclosure is not limited to these three color channels (in principle, any number and type of color channels can be selected). Input image data 1 is input into a neural network, such as a CNN, to generate output multispectral image data, such as multispectral image data 2. In this embodiment, output multispectral image data 2 is generated from input image data 1, and the number of spectral channels in output multispectral image data 2 is nine (9). Therefore, input image data represented by multiple color channels has been converted into output multispectral image data 2 represented by multiple spectral channels.

[0033] The neural network generates at least first and second multispectral image data based on input image data, wherein the second multispectral image data has a greater number of spectral channels than the first multispectral image data. The neural network can also generate multiple multispectral image data based on input image data. That is, each of the multiple multispectral image data can be followed by another multispectral image data, and each of the multiple multispectral image data can be generated based on previously generated multispectral image data. Therefore, multiple intermediate multispectral image data can be generated by the neural network.

[0034] In some embodiments, the circuit may also be configured to acquire first or second multispectral image data as output multispectral image data. As described above, the neural network can generate at least first and second multispectral image data; therefore, based on, for example, user settings or predetermined settings of the image processing device for the target application, the circuit can acquire first or second multispectral image data as output multispectral image data.

[0035] In some embodiments, the input image data may include spectral image data. For example, spectral image data may be input image data represented by a small number of spectral channels, which may be suitable for applications such as object classification using neural networks. The input image data may also include red-green-blue (RGB) image data represented by a specific number of color channels, wherein multiple spectral channels are integrated, as described above.

[0036] In some embodiments, the number of spectral channels in the output multispectral image data is greater than the number of spectral channels in the input image data. When multispectral image data represented by a small number of spectral channels (e.g., six (6) spectral channels) is input to the processing device, the number of spectral channels in the output multispectral image data can be nine (9), etc. Therefore, the output multispectral image data can have a higher spectral resolution after processing.

[0037] Typically, it is desirable for the size of the image data to remain the same before and after image processing, even when the output image data has a higher spectral resolution after image processing. Therefore, in some embodiments, the spatial resolution of the first multispectral image data can be higher than that of the second multispectral image data.

[0038] As mentioned above, conventional imaging devices using traditional neural networks (e.g., mosaic array multispectral imaging devices) typically sacrifice spatial resolution for spectral resolution, both of which offer benefits for computational sensing applications. Therefore, generating multispectral images from RGB images or from multispectral images represented by a small number of spectral channels may be appropriate, offering an optimized trade-off between spatial and spectral resolution for the device.

[0039] Therefore, in some embodiments, output multispectral image data is generated based on a predetermined relationship between spatial resolution and the number of spectral channels. This predetermined relationship can be an optimal trade-off between spectral resolution and spatial resolution. The optimal point of this trade-off may depend on the system, application, target scene, etc. The predetermined relationship between spatial resolution and the number of spectral channels can be determined based on user settings or predetermined settings of the image processing device for the target application.

[0040] Figure 2 An exemplary optimization relationship between spectral resolution and spatial resolution of multispectral image data (e.g., multispectral skin data) is illustrated. Specifically, the number of spectral channels (e.g., spectral bands) is represented by vertical bars on the x-axis. In this embodiment, the number of spectral bands increases from three (3) to three hundred (300). Classification accuracy is represented by the y-axis, and in this embodiment, classification accuracy increases to sixteen (16) spectral bands as the number of spectral bands increases. The dashed line 3 represents the spatial resolution of the image. In this case, optimized performance is achieved using multispectral data with 16 spectral channels. As mentioned above, the optimal relationship between spectral resolution and spatial resolution depends on the objective. For example, the optimal trade-off may differ for other applications. Furthermore, the optimal relationship between spectral resolution and spatial resolution may vary depending on the content of the scene, which can make it difficult to design an optimal multispectral sensor with the best performance.

[0041] In some embodiments, the neural network may be a convolutional neural network (CNN), without limiting the scope of this disclosure. For example, in some embodiments, the convolutional neural network may include convolutional layers, or may also include local or global pooling layers, such as maximal pooling layers, which reduce the dimensionality of the image data, as is well known.

[0042] Pooling layers can be used for pooling, which is a form of nonlinear downsampling, such as spatial pooling, i.e., max pooling, average pooling, sum pooling, etc.

[0043] The generation of multispectral image data can occur during the training phase of a neural network (e.g., a CNN), or it can be generated using a pre-trained neural network (e.g., a trained CNN), for example, to extract information from the image data (e.g., object recognition or recognition of other information in the image data, such as spatial information, spectral information, patterns, colors, etc.). Therefore, the neural network can be an untrained neural network.

[0044] Furthermore, the neural network can be part of the image processing device, for example, stored in the memory or RAM of the image processing device, or the image processing device can access the neural network, for example, via inter-processor communication, electronic bus, network (including the Internet), etc.

[0045] exist Figure 3 The diagram illustrates the general principles used in CNNs, with the first row showing the overall CNN structure and the second row showing the basic principles of the building blocks. The principles of CNNs and their applications in imaging are well-known; therefore, the following explanation is based solely on... Figure 3 Let's discuss it briefly.

[0046] The input image includes, for example, three maps or layers (exemplary red, green, and blue (RGB) color information) and N×N blocks. A CNN has convolutional layers and subsequent pooling layers, where this structure can be repeated, as shown below. Figure 3 As shown. Convolutional layers consist of neurons. By applying kernels (filters) (see convolutional kernels in the second row) to the input image, corresponding feature maps can be obtained. In this embodiment, information about the most active neurons in the convolutional layer is obtained based on the maximum pooling layer (see "Maximum Pooling" in the second row), and other information is discarded. In several repetitions ( Figure 3 After three iterations (into the third), the process ends with a fully connected layer, also known as an affine layer. This final layer typically contains multiple neurons, corresponding to the number of object categories (output features) that the CNN will distinguish. The output is... Figure 3 The first row shows the output distribution, which is represented by a row of columns, where each column represents a category, and the column height represents the weight of the object category. Different categories correspond to output or image attribute features from the CNN. For example, these categories are "people, cars, etc." Typically, hundreds or thousands of categories can be used, for example, for object recognition of different objects.

[0047] In some embodiments, a convolutional neural network (CNN) can be trained to generate first multispectral image data from input image data and to generate second multispectral image data from the first multispectral image data. As described above, a CNN can generate multiple multispectral image data sets, such as first and second multispectral image data sets, which are generated based on and follow the first multispectral image data. That is, each of the multiple multispectral image data sets can be generated based on previously generated multispectral image data, and each of the multiple multispectral image data sets can be followed by another multispectral image data set.

[0048] In some embodiments, a convolutional neural network (CNN) can be trained based on RGB image data and multispectral image data. Training data for multispatial multispectral images can also be generated from high-resolution hyperspectral data (the terms multispectral and hyperspectral data are well known in the art and are typically distinguished by the number of spectral channels, with hyperspectral data having more spectral channels than multispectral data). Typically, in image processing, CNNs use ground-based image data and desired image data as a training database, such as RGB image data and multispectral image data.

[0049] Specifically, multispectral image data represented by C channels is generated from hyperspectral image data using the following equation:

[0050]

[0051] Among them, I c S is the intensity of the spectral band c (spectral channel) of the multispectral image, λ is the integrated wavelength, R is the spectral reflectance of the target in the scene, L is the spectral distribution of the illumination, for example, white illumination, which has a flat spectral distribution across all wavelengths. c is the sensor's spectral sensitivity in the spectral band c, and n is the sensor noise.

[0052] Here, R is the hyperspectral data (HS image) measured by a hyperspectral camera, L can be set to take into account the illumination to be used in the application, and S... c This is given by the camera's sensor specifications.

[0053] In some embodiments, the circuitry is also configured to perform object recognition. For example, object recognition can be performed in autonomous vehicle applications, where the size of a pedestrian in an image can depend on their distance from the vehicle. To detect pedestrians far from the vehicle, images with higher spatial resolution may be suitable for the pedestrian detector. Furthermore, object recognition can be performed, for example, in hand recognition applications, where hands can make various gestures. In this case, spatial resolution is less useful than spectral resolution. Therefore, the relationship between spectral and spatial resolution can include more spectral information than just spatial information.

[0054] Furthermore, image processing based on multispectral and hyperspectral imaging is widely used in the food industry (e.g., fruit abrasion detection, fish freshness detection), material classification applications (e.g., remote sensing, medical diagnosis), etc., and therefore, some embodiments belong to these fields.

[0055] Some embodiments relate to image processing methods that can be performed by the image processing apparatus described herein or any other electronic device, processor, or other computing device. The method includes acquiring input image data represented by a plurality of color channels and feeding the input image into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein the second multispectral image data has a greater number of spectral channels than the first multispectral image data.

[0056] The image processing method may further include acquiring first or second multispectral image data as output multispectral image data. As described above, the input image data may include spectral image data, wherein the number of spectral channels in the output multispectral image data may be greater than the number of spectral channels in the input image data. Furthermore, the spatial resolution of the first multispectral image data may be higher than the spatial resolution of the second multispectral image data. The output multispectral image data may be generated based on a predetermined relationship between spatial resolution and the number of spectral channels. Additionally, the neural network may be a convolutional neural network, which can be trained to generate the first multispectral image data from the input image data and to generate the second multispectral image data from the first multispectral image data. Furthermore, the convolutional neural network may also be trained based on RGB image data and multispectral image data, as discussed herein. The image processing method may also include performing object recognition.

[0057] Return to reference Figures 4 to 8 The embodiments are described below with reference to Figure 4 Examples of image processing apparatuses are discussed.

[0058] exist Figure 4The diagram shows a block diagram of an embodiment of an image processing apparatus 11 that inputs image data into a convolutional neural network (CNN) to generate multispectral image data, as described herein.

[0059] In this embodiment, the image processing device 11 includes a circuit 12 having an interface 13, a central processing unit (CPU) 14 including a plurality of processors including a graphics processing unit (GPU), a memory 15 including RAM, ROM and memory, and a trained CNN 16 (stored in the memory).

[0060] Image processing device 11 acquires image data through interface 13, for example, input image data 1, which is represented by multiple color channels, namely red, green and blue in this embodiment. Input image data 1 represents an image of a target scene captured by a digital camera such as an RGB camera (not shown).

[0061] Input image data 1, represented by multiple color channels, is transmitted to CPU 14. CPU 14 feeds input image data 1 into CNN 16 to generate multispectral image data represented by multiple spectral channels. CNN 16 has been pre-trained to generate (at least) first and second multispectral image data based on input image data 1. As discussed herein, image processing device 11 is configured to acquire either the first or second multispectral image data generated by CNN 16 as output multispectral image data, for example, output multispectral image data 2.

[0062] In this embodiment, the image processing device 11 acquires second multispectral image data and outputs it as multispectral image data 2. The number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data. For example, the number of spectral channels in the second multispectral image data is nine (9).

[0063] The implementation of the image processing device 11 described above can lead to a reduction in computational workload and memory usage. Furthermore, the CNN 16 can be a single CNN capable of generating multi-spatial, multispectral image data from RGB images.

[0064] In the following text, see references Figure 5 This explains the use of CNN 16 to generate multiple multispectral image data based on input image data 1.

[0065] Figure 5 An embodiment of a processing scheme for a learning method of a CNN16 for generating multiple multispectral image data 20-1 to 20-N based on input image data 1 is shown, wherein each of the multiple multispectral image data can be acquired from the image processing device 11 as... Figure 4The output is multi-spectral image data 2.

[0066] The image processing device 11 inputs the input image data 1 into the CNN 16. For example, the RGB image data, which is represented by multiple color channels, namely red, green, and blue. As mentioned before, the input image in the CNN has a shape, that is, (number of images) x (image width) x (image height) x (image depth). In this embodiment, the input image data 1 represents the input image, and its height and width define the spatial resolution. The height of the input image data 1 is Height0, and the width is Width0. The number of spectral channels of the input image data 1 is Ch0. The convolutional layer of the CNN 16 performs convolution on the input image data 1, uses the rectified linear unit (RELU) and spatial pooling performed by the max pooling layer to perform correction, and then passes the result to the next layer. The result of the next layer is the multi-spectral image data 20-1 represented by six (6) spectral channels (for example, corresponding to the first multi-spectral image data), and the multi-spectral image data 20-1 represents the multi-spectral image, which has a height Height1, a width Width1, and multiple spectral channels Ch1, where Height0>Height1, Width0>Width1, and Ch0<Ch1. Therefore, the result of the next layer is the multi-spectral image data 20-2 represented by nine (9) spectral channels (for example, corresponding to the second multi-spectral image data), and the multi-spectral image data 20-2 represents the multi-spectral image, which has a height Height2, a width Width2, and multiple spectral channels Ch2, where Height0>Height1>Height2, Width0>Width1>Width2, and Ch0<Ch1<Ch2. In this embodiment, the convolutional process is performed as described above until the size of the multi-spectral image data becomes the size of the kernel of the CNN 16. The result of the last layer of the CNN 16 is the multi-spectral image data 20-N represented by twelve (12) spectral channels (for example, corresponding to the Nth multi-spectral data), and the multi-spectral image data 20-N represents the multi-spectral image, which has a height Height N , width Width N and multiple spectral channels Ch N , where Height0>Height1>Height2>…>Height N , Width0>Width1>Width2>…>Width N and Ch0<Ch1<Ch2<…<Ch N .

[0067] The CNN 16 is trained so that the image processing device 11 can acquire any one of the multispectral image data 20-1 to 20-N (e.g., first to Nth multispectral data) as output multispectral image data 2. That is, the CNN 16 generates multiple intermediate multispectral image data at several points in the neural network. Furthermore, the image processing device 11 acquires any one of the multispectral image data 20-1 to 20-N with a predetermined relationship between spatial resolution and spectral resolution, depending on the application or target scene. This predetermined relationship can be pre-set by the user; therefore, when the predetermined relationship (an optimized relationship between spatial resolution and spectral resolution) is achieved, the CNN 16 no longer performs computation.

[0068] The above embodiments do not limit this disclosure in this respect. For example, a suitable multispectral image can be determined by analyzing the spatial frequency distribution of the input RGB image. Depending on the application being performed, such as object classification using a CNN, a multispectral image with a small number of spectral channels may be desirable. On the other hand, spectral information may be more important for the application being performed, and in some embodiments, a multispectral image with a large number of spectral channels may be desirable. Furthermore, the target performance can be determined by the results of the application, such as the reliability of the object classification results. For example, when a multispectral image with a small number of spectral channels is input, the application results may not meet the target performance; therefore, the CNN may continue to generate multispectral images with a large number of spectral channels until the application results meet the set criteria.

[0069] exist Figure 6 An embodiment of the learning system 30 is shown in the block diagram, which generates a learning model and trains a neural network, such as CNN 16, based on the learning model.

[0070] The learning system 30 includes a storage device (e.g., a reference). Figure 4 The image processing apparatus 11 described includes a memory 15, an RGB image generator 31, a multispectral image generator 32, a learning device (e.g., a CNN 16), and a learning model 33.

[0071] A high-resolution hyperspectral image, represented by hyperspectral (FIS) image data, is stored in the memory 15 of the image processing apparatus 11, having an image resolution of (H)ight*(W)idth*(C)channels. Then, an RGB image with a resolution h0(≤H)*w0(≤W)*c0(=3) is generated from the hyperspectral (HS) image by the RGB image generator 31, and multiple MS images are generated from the HS image by the multispectral image generator 32, the MS images having resolutions such as h... i ( <h0)*w i ( <w0)*c iThe resolution is (>c0), where i = 1, 2, ..., N, and N is the number of intermediate MS images, for example, represented by multiple multispectral image data, for example... Figure 5 Multispectral image data 20-1 to 20-N. In this embodiment, the intermediate MS images should meet the following conditions:

[0072] (a)h i *w i *c i =h0*w0*c0

[0073] (b)h i h i +l,w i >w i +i,c i <c i +1

[0074] If already Figure 5 As described in detail. Learning devices such as CNN 16 are trained to transform multiple MS images from an RGB image by minimizing the following reconstruction loss: Loss = MSE(MS REC1 -MS GT1 )+MSE(MS REC2 -MS GT2 )+…+MSE(MS REC(N) -MS GT(N) ).

[0075] Among them, MS REC(i) MS images reconstructed by CNN i MS GT(i) MS image generated by multispectral image generator 32 i The basic fact is that MSE is the mean squared error function, and this disclosure is not limited in this respect. The mean absolute error function, etc., can also be used. The learning system 30 generates a learning model, which is stored in the memory 15 of the image processing device 11.

[0076] Figure 7 An embodiment of an image processing system 40 is shown, and the figure illustrates a block diagram of the image processing system 40.

[0077] The image processing system 40 includes an image capture device 41 (e.g., a camera including an RGB image sensor), an image processing apparatus 11 for storing a database, and an information processing apparatus 44 including a target region detection unit 45, a feature extraction unit 46, and a recognition unit 47.

[0078] The image processing system 40 is configured to perform object recognition on image data provided by the image capturing device 41 and processed by the image processing apparatus 11.

[0079] In this embodiment, an image capture device 41, such as an RGB camera, captures an image of the target scene, for example, an RGB image, and sends RGB image data representing the captured RGB image to an image processing device 11. The image processing device 11 outputs multispectral image data 43, which is generated by a trained convolutional neural network (e.g., CNN 16) trained based on a learning model 33. The generated multispectral image data 43 is also generated based on input information 42 related to the target to be identified. The generated multispectral image data 43 is transmitted to an information processing device 44, which is configured to perform object recognition. Hereinafter, regarding the object recognition performed by the information processing device 44, the multispectral image data 43 is transmitted to a target region detection unit 45, a feature extraction unit 46, and then to a recognition unit 47. The recognition unit 47 performs object recognition based on data included in a database stored in a memory 48. The output 49 of the image processing system 40 depends on the target recognition result (e.g., user ID).

[0080] In the following text, see references Figure 8 The image processing method 50 performed by the image processing device 11 and / or the image processing system 40 in some embodiments is discussed.

[0081] At 51, as described above, input image data, for example, input image data 1, is acquired by the image processing device 11 and / or the image processing system 40.

[0082] Input image data can be obtained from an image sensor, from memory included in the device, from external memory, or from an artificial image generator created via computer-generated graphics.

[0083] At 52, the input image data is fed into a convolutional neural network, such as CNN 16, to generate output multispectral (MS) image data at 53, such as output multispectral image data 2, as described above.

[0084] The input image data can be represented by multiple color channels, such as red, green and blue, or by a small number of spectral channels (e.g., three (3) channels), and so on.

[0085] In step 54, the convolutional neural network generates first and second multispectral image data based on the input image data.

[0086] As described above, the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data.

[0087] At 55, acquire the first or second multispectral image data as the output multispectral data.

[0088] As described in this article, a first multispectral image data is generated based on the input image data, and a second multispectral image data is generated based on the first multispectral image data.

[0089] At 56, output the acquired multispectral image data.

[0090] It should be understood that the embodiments describe an exemplary ordering of method steps. However, the specific order of the method steps is given for illustrative purposes only and should not be construed as binding.

[0091] The methods described herein are also implemented as computer programs in some embodiments, which, when executed on a computer and / or processor, cause the computer and / or processor to perform the methods. In some embodiments, a non-transitory computer-readable recording medium in which a computer program product is stored, which, when executed by a processor (e.g., the processor described above), causes the execution of the methods described herein.

[0092] Unless otherwise stated, all units and entities described in this specification and claimed in the appended claims may be implemented as integrated circuit logic, for example, on a chip, and unless otherwise stated, the functionality provided by such units and entities may be implemented by software.

[0093] With regard to the implementation of the above-disclosed embodiments using at least part of a software-controlled data processing device, it should be understood that providing such a software-controlled computer program and the transmission, storage or other medium providing such a computer program are contemplated as aspects of this disclosure.

[0094] Note that this technology can also be configured as described below.

[0095] (1) An image processing apparatus, comprising a circuit configured to:

[0096] Acquire input image data represented by multiple color channels;

[0097] Input image data is fed into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data.

[0098] (2) The image processing apparatus according to (1), wherein the circuit is further configured to acquire first or second multispectral image data as output multispectral image data.

[0099] (3) The image processing apparatus according to (1) or (2), wherein the input image data includes spectral image data.

[0100] (4) The image processing apparatus according to (3) wherein the number of spectral channels of the output multispectral image data is greater than the number of spectral channels of the input image data.

[0101] (5) The image processing apparatus according to any one of (1) to (4), wherein the spatial resolution of the first multispectral image data is higher than the spatial resolution of the second multispectral image data.

[0102] (6) The image processing apparatus according to (5) wherein output multispectral image data is generated based on a predetermined relationship between spatial resolution and the number of spectral channels.

[0103] (7) The image processing apparatus according to any one of (1) to (6), wherein the neural network is a convolutional neural network.

[0104] (8) The image processing apparatus according to (7), wherein a convolutional neural network is trained to generate first multispectral image data from input image data and to generate second multispectral image data from the first multispectral image data.

[0105] (9) The image processing apparatus according to (7), wherein a convolutional neural network is trained based on RGB image data and multispectral image data.

[0106] (10) The image processing apparatus according to any one of (1) to (9), wherein the circuitry is further configured to perform object recognition.

[0107] (11) An image processing method, comprising:

[0108] Acquire input image data represented by multiple color channels;

[0109] Input image data is fed into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least first and second multispectral image data based on the input image data, wherein the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data.

[0110] (12) The image processing method according to (11) further includes acquiring first or second multispectral image data as output multispectral image data.

[0111] (13) The image processing method according to (11) or (12), wherein the input image data includes spectral image data.

[0112] (14) According to the image processing method described in (13), the number of spectral channels of the output multispectral image data is greater than the number of spectral channels of the input image data.

[0113] (15) The image processing method according to any one of (11) to (14), wherein the spatial resolution of the first multispectral image data is higher than the spatial resolution of the second multispectral image data.

[0114] (16) According to the image processing method described in (15), output multispectral image data is generated based on a predetermined relationship between spatial resolution and the number of spectral channels.

[0115] (17) The image processing method according to any one of (11) to (16), wherein the neural network is a convolutional neural network.

[0116] (18) The image processing method according to (17), wherein a convolutional neural network is trained to generate first multispectral image data from input image data and to generate second multispectral image data from the first multispectral image data.

[0117] (19) The image processing method according to (17) wherein a convolutional neural network is trained based on RGB image data and multispectral image data.

[0118] (20) The image processing method according to any one of (11) to (19) further includes performing object recognition.

[0119] (21) A computer program comprising program code that, when executed on a computer, causes the computer to perform the method according to any one of (11) to (20).

[0120] (22) A non-transitory computer-readable recording medium in which a computer program product is stored, which, when executed by a processor, causes the execution of the method according to any one of (11) to (20).

Claims

1. An image processing apparatus, comprising circuitry configured to: Acquire input image data represented by multiple color channels; The input image data is fed into a neural network to generate output multispectral image data, wherein... The neural network is configured to generate at least a first multispectral image data and a second multispectral image data based on the input image data, wherein the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data. Analyze the spatial frequency of the input image data; Based on the analyzed spatial frequencies, a predetermined relationship is determined between the spatial resolution and the number of spectral channels of the output multispectral image data; and Based on the predetermined relationship, the output multispectral image data is obtained from the generated first or second multispectral image data.

2. The image processing apparatus according to claim 1, wherein, The input image data includes spectral image data.

3. The image processing apparatus according to claim 2, wherein, The number of spectral channels in the output multispectral image data is greater than the number of spectral channels in the input image data.

4. The image processing apparatus according to claim 1, wherein, The spatial resolution of the first multispectral image data is higher than that of the second multispectral image data.

5. The image processing apparatus according to claim 1, wherein, The neural network in question is a convolutional neural network.

6. The image processing apparatus according to claim 5, wherein, The convolutional neural network is trained to generate the first multispectral image data from the input image data, and to generate the second multispectral image data from the first multispectral image data.

7. The image processing apparatus according to claim 5, wherein, The convolutional neural network is trained based on RGB image data and multispectral image data.

8. The image processing apparatus according to claim 1, wherein, The circuit is also configured to perform object recognition.

9. An image processing method, comprising: Acquire input image data represented by multiple color channels; The input image data is input into a neural network to generate output multispectral image data, wherein the neural network is configured to generate at least a first multispectral image data and a second multispectral image data based on the input image data, wherein the number of spectral channels in the second multispectral image data is greater than the number of spectral channels in the first multispectral image data. Analyze the spatial frequency of the input image data; Based on the analyzed spatial frequencies, a predetermined relationship is determined between the spatial resolution and the number of spectral channels of the output multispectral image data; and Based on the predetermined relationship, the output multispectral image data is obtained from the generated first or second multispectral image data.

10. The image processing method according to claim 9, wherein, The input image data includes spectral image data.

11. The image processing method according to claim 10, wherein, The number of spectral channels in the output multispectral image data is greater than the number of spectral channels in the input image data.

12. The image processing method according to claim 9, wherein, The spatial resolution of the first multispectral image data is higher than that of the second multispectral image data.

13. The image processing method according to claim 9, wherein, The neural network in question is a convolutional neural network.

14. The image processing method according to claim 13, wherein, The convolutional neural network is trained to generate the first multispectral image data from the input image data, and to generate the second multispectral image data from the first multispectral image data.

15. The image processing method according to claim 13, wherein, The convolutional neural network is trained based on RGB image data and multispectral image data.

16. The image processing method according to claim 9 further includes performing object recognition.

Images