Method and apparatus for selecting waveband and method for generating spectral image processing model

Abstract

The present disclosure relates to a method and apparatus for selecting wavebands and a method for generating a spectral image processing model. According to one embodiment of the present disclosure, the method for selecting wavebands comprises: generating a first waveband set similarity matrix representing similarities between each waveband in N wavebands based on a first two-dimensional convolution layer of a first spectral image processing model trained based on first-class objects as training samples using spectral responses corresponding to the first waveband set; and selecting n wavebands from the N wavebands as a second waveband set associated with a second spectral image processing model based on at least the first waveband set similarity matrix; wherein N>n. The method and apparatus of the present disclosure have at least one of the following benefits: reducing computational cost, improving accuracy performance of the model, reducing acquisition time of test data, and reducing test data acquisition cost.

Description

Technical Field

[0001] This disclosure relates generally to spectral image processing, and more specifically to methods for selecting bands, methods for generating spectral image processing models, and apparatus for selecting bands in spectral image processing. Background Technology

[0002] Spectral images contain the spectral response of each pixel across all bands. Spectral images can be categorized as multispectral, hyperspectral, and ultra-hyperspectral. Unlike ordinary RGB images, which only capture red, green, and blue visible light, hyperspectral (HSI) images capture information from dozens or even hundreds of spectral bands across a larger spectral range. Hyperspectral images have higher spectral resolution, providing more and more accurate object information than RGB images, better reflecting the object's properties. Therefore, HSI is widely used in remote sensing and histopathological imaging. Image classification, segmentation, and object recognition using HSI are more accurate than with RGB images. In recent years, in the field of deep learning involving artificial intelligence, methods using neural network models to train and test HSI data have become increasingly common, showing promising application prospects.

[0003] While HSI can provide more and more accurate object information, significantly improving the performance of subsequent algorithms, it may also contain a large number of redundant and noisy band images. These band images not only fail to benefit the development of subsequent algorithms but often lead to performance degradation and increased training difficulty. Therefore, HSI-based band selection algorithms have always been a research hotspot in the field of HSI research.

[0004] Specifically, the band selection method has the following advantages: 1. It removes redundant or noisy band images; 2. It significantly reduces the training and inference time of subsequent algorithms, lowers the training difficulty of the algorithms, and avoids the adverse effects of dimensionality explosion; 3. It significantly reduces the complexity, cost, and acquisition time of the acquisition camera, which is conducive to the promotion and visualization of HSI; 4. Since redundant and noisy bands are removed, the training difficulty is reduced, and the selection algorithm of a very small number of high-performance bands is conducive to improving the accuracy and performance of subsequent algorithms.

[0005] With the popularization of deep learning methods, band selection methods have increasingly been involved in deep learning approaches in the past five or six years. More and more band selection methods have been proposed for selecting bands for deep neural networks. Among them, the band selection method based on neural network pruning (hereinafter referred to as "Method 2") was first proposed in 2020 in the following reference 1, achieving excellent performance results: Reference 1: Wang, Qixiong, Xiaoyan Luo, and Jihao Yin, "Neural Network Pruning for Hyperspectral Image Band Selection", IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium, IEEE, 2020.

[0006] While Method 2 exhibits superior performance, it still suffers from several drawbacks. First, due to the L1 norm's lack of robustness and stability, models trained multiple times on the same data will exhibit slight differences, which can lead to completely different band sets selected by this method. Second, it exhibits low discrimination ability against noisy bands. Finally, it fails to consider the similarity between bands during selection, resulting in band clustering and excessive redundancy, which degrades the performance of retrained models after selection. Furthermore, for the intended spectral image processing task, using spectral images with spectral responses containing fewer bands is desirable. Summary of the Invention

[0007] A brief overview of this disclosure is provided below to offer a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0008] According to one aspect of this disclosure, a method for selecting bands for spectral image processing is provided. The method includes: generating a first band set similarity matrix characterizing the similarity between bands in the N bands based on a first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects as training samples, which contains spectral responses corresponding to a first band set of N bands; and selecting n bands from the N bands as a second band set associated with a second spectral image processing model, at least based on the first band set similarity matrix; wherein N>n; and the second spectral image processing model is configured to perform spectral image processing on a second class of objects containing spectral responses corresponding to the second band set.

[0009] According to one aspect of this disclosure, a method for generating a spectral image processing model is provided. The method includes: determining a second band set using the aforementioned band selection method; and generating the spectral image processing model by training a second spectral image processing model using a second class of objects containing spectral responses corresponding to the second band set.

[0010] According to one aspect of this disclosure, an apparatus for selecting bands for spectral image processing is provided. The apparatus includes: a memory storing instructions; and at least one processor capable of communicating with the memory to execute instructions retrieved from the memory, the instructions causing the at least one processor to: generate a first band set similarity matrix characterizing the similarity between bands in the N bands, based on a first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects as training samples containing spectral responses corresponding to a first band set of N bands; and select n bands from the N bands as a second band set associated with a second spectral image processing model, at least based on the first band set similarity matrix; wherein N>n; and the second spectral image processing model is configured to perform spectral image processing on a second class of objects containing spectral responses corresponding to the second band set.

[0011] According to one aspect of this disclosure, a computer-readable recording medium storing a program is provided. When executed, the program causes a computer to function as: a generation unit for generating a first band set similarity matrix characterizing the similarity between bands in the N bands, based on a first two-dimensional convolutional layer of a first spectral image processing model trained using spectral responses corresponding to a first band set of N bands as training samples of a first class of objects; and a selection unit for selecting n bands from the N bands as a second band set associated with a second spectral image processing model, at least based on the first band set similarity matrix; wherein N>n; and the second spectral image processing model is configured to perform spectral image processing on a second class of objects containing spectral responses corresponding to the second band set.

[0012] The beneficial effects of the methods, apparatus, and storage media disclosed herein include at least one of the following: reducing redundant bands, fewer noisy bands, reducing model complexity, reducing computational costs, improving model accuracy performance, reducing test data acquisition time, and reducing test data acquisition costs. Attached Figure Description

[0013] The embodiments of this disclosure are described below with reference to the accompanying drawings, which will help to more easily understand the above and other objects, features, and advantages of this disclosure. The drawings are only for illustrating the principles of this disclosure. The dimensions and relative positions of the elements are not necessarily drawn to scale in the drawings. The same reference numerals may denote the same features. In the drawings:

[0014] Figure 1 An exemplary flowchart of a method for selecting a band for spectral image processing according to an embodiment of the present disclosure is shown;

[0015] Figure 2 An exemplary schematic diagram of a first spectral image processing model based on a neural network is shown;

[0016] Figure 3 An exemplary flowchart of a method for generating a spectral image processing model according to an embodiment of the present disclosure is shown;

[0017] Figure 4 The L1 norm of each band in the experiment is shown;

[0018] Figure 5 A schematic diagram of the similarity matrix of the first band set in the experiment is shown;

[0019] Figure 6 The results of the comparative experiment are shown;

[0020] Figure 7 An exemplary block diagram of an apparatus for selecting a band according to an embodiment of the present disclosure is shown;

[0021] Figure 8 An exemplary block diagram of an apparatus for generating a spectral image processing model according to an embodiment of the present disclosure is shown;

[0022] Figure 9 An exemplary block diagram of an apparatus for selecting a band according to an embodiment of the present disclosure is shown; and

[0023] Figure 10 This is an exemplary block diagram of an information processing device according to one embodiment of the present disclosure. Detailed Implementation

[0024] Exemplary embodiments of this disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0025] It should also be noted that, in order to avoid obscuring the contents of this disclosure with unnecessary details, only the device structure closely related to the solution according to this disclosure is shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0026] It should be understood that this disclosure is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0027] Computer program code used to perform the operations of various aspects of embodiments of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0028] The method disclosed herein can be implemented using circuitry with corresponding functional configurations. The circuitry includes circuitry for a processor.

[0029] This disclosure generally relates to spectral image processing models (sometimes simply referred to as "models"). A spectral image processing model performs spectral image processing on an input object and outputs the processing result. The input object (also simply referred to as an "object") contains the spectral response corresponding to a band set with multiple bands. Here, "band" corresponds to a spectral band, which can be represented as a wavelength range centered at a certain wavelength. That is, each spectral image processing model corresponds to a corresponding band set. This disclosure includes the following approach: the first two-dimensional convolutional layer of a model based on a corresponding band set with a larger number of bands selects a corresponding band set with a smaller number of bands, and generates a model corresponding to the subsequent corresponding band set through training. The latter model may have advantages in terms of cost and performance. For distinction, the terms "first band set," "first type of object," "first type of spectral image," and "first spectral image processing model" are used for scenarios before the number of bands changes; the terms "second band set," "second type of object," "second type of spectral image," and "second spectral image processing model" are used for scenarios after the number of bands changes.

[0030] One aspect of this disclosure provides a method for selecting bands for spectral image processing. For a spectral image, the spectral response for a particular band can be approximated as the spectral response for the corresponding center wavelength. The spectral response is, for example, spectral intensity. A spectral image containing a band set of multiple bands contains spectral responses corresponding to that band set. Spectral image processing models can perform image processing such as localization and recognition based on these spectral responses. Hyperspectral images have a narrower wavelength range in their bands, thus exhibiting higher spectral resolution. This band selection method can be implemented using a computer. (See below for reference.) Figure 1 and Figure 2 The method is described exemplarily below.

[0031] Figure 1 An exemplary flowchart of a method 100 for selecting bands for spectral image processing according to one embodiment of the present disclosure is shown. Method 100 relates to a first two-dimensional convolutional layer of a first spectral image processing model for processing a first type of object comprising N bands of spectral response. Figure 2 An exemplary schematic diagram of a first spectral image processing model MN1 based on a neural network is shown. Model MN1 includes an input layer Lin, an output layer Lut, and multiple two-dimensional convolutional layers, including a first two-dimensional convolutional layer L1st. The input of model MN1 is a first type of object containing spectral responses corresponding to a first band set of N bands. This first type of object may correspond to a first type of spectral image containing spectral responses corresponding to the first band set of N bands, or an image patch within the aforementioned spectral image. For example, the spectral response sequence I1, ..., I20 ... i ... I N There is a correspondence with the first band set {band 1, ..., band i, ..., band N}, where I represents the spectral intensity, and index i represents any band i among the N acquisition bands for the first type of object. Band i, for example, corresponds to a certain wavelength range. Figure 2 The document also illustrates two terms involved in the band selection method of this disclosure: “Lp norm” and “first band set similarity matrix Mats”, where Lp and Mats are derived based on the first two-dimensional convolutional layer L1st. Further details of Lp and Mats will be described below.

[0032] Method 100 includes steps S101 and S103. In step S101, based on the first two-dimensional convolutional layer L1st of a first spectral image processing model MN1 trained using a first class of objects ObN, which contain spectral responses corresponding to a first band set SBN with N bands as training samples, a first band set similarity matrix Mats is generated, representing the similarity between bands in the N bands. N is a natural number, for example, N = 128. In other words, the spectral dimension of the first class of objects is N, i.e., it contains spectral responses of N bands. The first class of objects, for example, corresponds to a first class of spectral images such as hyperspectral images. The first class of spectral images, for example, have high spectral resolution. Model MN1 is a model obtained by training an initial model MN0 using the first class of objects ObN. Models MN0 and MN1 are neural network-based models. The number of samples used to train model MN0 is one or more. To ensure the accuracy performance of model MN1, a large number of samples can be used to train model MN0 in an iterative manner. A spectral image can be viewed as a sequence of multiple two-dimensional images arranged along the spectral dimension. These two-dimensional images are aligned with each other in two-dimensional spatial coordinates, and each image represents the scene's response at each acquired spectral band; therefore, each two-dimensional image is also called a band image. Depending on the model's requirements for the input object, a spectral image can be used as one sample or multiple samples. Each training sample contains the spectral response of each band i in N bands, with a resolution of H*W, where H represents height and W represents width. In one example, N equals the number of bands acquired (captured). Model MN1 is configured, for example, to perform spectral image processing such as image classification, image segmentation, or object recognition on the spectral image ImN to be processed, where the spectral image ImN is a first-class spectral image containing the spectral responses corresponding to the aforementioned N bands. Figure 2 As shown, model MN1 performs aliasing on different band signals from the first type of object, such as a hyperspectral image, to extract features. The first band aliasing module of model MN1 is a two-dimensional convolutional layer. Figure 2 The first 2D convolutional layer (L1st) is shown in the diagram. Therefore, this 2D convolutional layer reflects the contribution of the corresponding spectral response for each band in the first class of objects (e.g., hyperspectral images) to the task. The first band set similarity matrix Mats generated by the first 2D convolutional layer L1st based on model MN1 can be determined based on the parameters of the first 2D convolutional layer L1st. The input to the first 2D convolutional layer L1st is the first class of objects including the spectral responses corresponding to N bands, i.e., N input channels and C output channels of feature image.

[0033] In step S103, at least based on the first band set similarity matrix Mats, n bands are selected from N bands as a second band set SBn associated with the second spectral image processing model Mn1; where N>n; and the second spectral image processing model Mn1 can extract features for spectral image processing based on a second type of object Obn containing the spectral response corresponding to the second band set SBn. That is, step S103 is a pruning operation, removing at least one band from the N bands, such that the number of selected bands is n. Figure 2 The example illustrates the removal of band 2. The selected n bands can be used to train or fine-tune a spectral image processing model. For example, a spectral image processing model Mn1 is obtained by training or fine-tuning a spectral image processing model Mn0 using only a second type of object (e.g., a second type of spectral image or an image patch of the second type of spectral image containing spectral responses corresponding to the second band set SBn) that contains only the spectral responses corresponding to the second band set SBn. Here, the initialization parameters of model Mn0 can be randomly generated, generated by a pre-trained model of other data, or extracted from a pre-trained model MN1. For example, model Mn0 can be obtained from a pre-trained MN1 by modifying it as follows: for the first two-dimensional convolutional layer L1st of MN1, only the parameters corresponding to the selected n bands are retained, while the parameters of other layers remain unchanged. Model Mn1 is used to process a spectral image, which is a spectral image to be processed containing spectral responses corresponding to the second band set SBn. Model Mn1 can perform spectral image processing on a second type of object containing spectral responses corresponding to a second band set. For example, it can extract features for spectral image processing from the spectral image Imn to be processed based on the selected n bands (i.e., the band set SBn). Model Mn1 can then determine relevant information about the spectral image Imn based on the extracted features, such as identifying objects in Imn. It can be understood that the components of the first band set similarity matrix Mats correspond to the N bands. Therefore, based on Mats, the differences between bands within various band groups selected from the N bands can be calculated, and the band group with the greatest difference can be selected as the chosen n bands, thereby solving problems such as excessive band redundancy.

[0034] The method for determining the similarity matrix Mats of the first band set is described in further detail below.

[0035] Each two-dimensional (2D) convolutional neural network contains a large number of 2D convolutional layers. Method 100 only involves the first 2D convolutional layer L1st of the convolutional neural network-based model MN1. The input to the first 2D convolutional layer L1st is a data sequence of N spectral responses corresponding to N bands, and the output is a feature image with C channels. The weight matrix of this convolutional layer... Where C is the number of output channels, N is the number of input channels, and H is the number of output channels. k *W k This refers to the kernel size of the 2D convolution in the 2D spatial domain. The weight coefficients corresponding to each input channel, i.e., the weight coefficients of the spectral response data corresponding to each band in the network, represent the contribution coefficient of the spectral response data of each band in the network. By analyzing this weight matrix, the inventors realized that if two bands have similar effects on spectral image processing, then their corresponding weights in the first convolutional layer should also be similar. Therefore, in this disclosure, the similarity between bands can be determined based on the corresponding weight vectors in the weight matrix Matw of the first two-dimensional convolutional layer L1st. For example, the component e in the first band set similarity matrix Mats... ij It can be calculated according to formula (1).

[0036] e ij =similarity(v i ,v j (1)

[0037] Among them, e ij It is the similarity between the i-th band and the j-th band, v i It is the component in the weight matrix Matw corresponding to the i-th band, v i It represents the component in the weight matrix Matw corresponding to the j-th band, and similarity() is the function for calculating similarity.

[0038] In one embodiment, the similarity between bands is determined based on the Euclidean or cosine distance between the corresponding weight vectors.

[0039] In one example, see Equation (2), similarity(v) can be determined based on the Euclidean distance. i v j ).

[0040] similarity(v i v j ) = 1 - Euclidean(v i v j ) / maxEuclidean (2)

[0041] Among them, Euclidean(v i v j ) is a vector v i v j The Euclidean distance between the two bands is given by equation (3). maxEuclidean is the maximum value of the Euclidean distance between the two weight vectors corresponding to any two bands for the matrix Matw.

[0042]

[0043] In one example, see equation (4), similarity(v) can be determined based on the cosine distance (cosine_distance). i v j ).

[0044] similarity(v i v j ) = 1 - cosine_distance(v i v j ) / twenty four)

[0045] Among them, cosine_distance(v i v j ) is a vector v i v j The cosine distance between them.

[0046] The following section provides a further description of the selection of n bands.

[0047] In one embodiment, given that the number of selected bands is n, it is desirable to select the n most dissimilar band combinations from the N bands (i.e., the first band set SBN) based on the first band set similarity matrix Mats. Preferably, the most dissimilar band combinations can be selected as the n selected bands from all band combinations containing n bands. Referring to equations (5) and (6), in one example, the degree of dissimilarity between the selected band combinations can be described using the matrix determinant, and the most dissimilar band combination c0 can be further selected as the second band set SBn.

[0048]

[0049] Among them, det(Mats) c ) is the matrix Mats c The determinant of the band combination c is shown in equation (6).

[0050]

[0051] That is, band combination c is a subset of bands containing n bands from the first band set SBN, which contains N bands. c This is the similarity matrix for band combination c, determined based on the similarity matrix Mats of the first band set. Specifically, the corresponding k1, k2, ..., k are extracted from the similarity matrix Mats. n rows and k1, k2, ..., k n Columns, forming a new matrix Matsc The matrix has dimensions n*n. When the band combination c is combination c0, log(det(Mats) c The maximum similarity is found by using an N*N similarity matrix to find a band subset with the greatest similarity difference. This band subset has a dimension of n.

[0052] Equation (5) can be solved using the method described in Reference 2:

[0053] Document 2: Chen, Laming, Guoxin Zhang, and Hanning Zhou. "Fast greedy mapinference for determinantal point process to improve recommendation diversity", Proceedings of the 32nd International Conference on NeuralSdormation Processing Systems, 2018.

[0054] In the solution process, this method requires manually selecting the first band k1 before a greedy algorithm can be used to greedily select the remaining bands. To achieve this, each band can be sorted according to its contribution to the network model, thus selecting the band with the highest contribution as the first band to be selected. Specifically, similar to reference 1, see equation (7), for any band i among the N bands, the Lp norm (p≥1) of the corresponding weights in the first 2D convolution can be calculated as the importance coefficient r of band i. i .

[0055]

[0056] In one embodiment, the importance coefficient of band i is calculated using the L1 norm. This importance coefficient reflects the contribution of each band to the model, and therefore, the most important band can be selected as the first band k1 according to equation (8).

[0057]

[0058] In one embodiment, selecting n bands from N bands based at least on a first band set similarity matrix includes: selecting n bands from N bands based on the first band set similarity matrix Mats and the importance coefficient r of each band in the N bands. In one example, equation (9) can be used to determine the band combination c1 of the selected n bands.

[0059]

[0060] Where θ∈[0,1], is a weighting coefficient used to adjust the weight of the band importance coefficient r in the optimization target norm. That is, n bands can be selected based on the weighted sum of the sum of the importance coefficients of some bands out of N bands and the logarithm of the determinant of the partial band similarity matrix of those bands. θ can be determined empirically. Tests show that, generally, the importance coefficient related to the Lp norm contributes little to band selection compared to the similarity matrix Mats; therefore, a smaller value for θ is preferred, for example, 0.0001 < θ < 0.2.

[0061] According to reference 2, the optimization problem corresponding to equation (9) can be equivalent to equation (10).

[0062]

[0063] Mats' c =Diag(exp(αr c ))·Mats c ·Diag(exp(αr c (11)

[0064] Where α = θ / [2(1-θ)], and Diag is a function for constructing the diagonal matrix. Similarly, the method in Reference 2 can be used to solve this optimization problem. This method greedily selects n bands as the chosen bands.

[0065] This disclosure also relates to a method for generating spectral image processing models. See below for details. Figure 3 An example is provided.

[0066] Figure 3An exemplary flowchart of a method 300 for generating a spectral image processing model according to an embodiment of the present disclosure is shown. In step S301, a second band set SBn is determined using a band selection method of the present disclosure (e.g., method 100). In step S303, a spectral processing model Mn1 is generated by training or fine-tuning a second spectral image processing model Mn0 using a second type of object containing spectral responses corresponding to the second band set SBn. Here, the training initialization parameters of model Mn0 can be randomly generated, generated by a pre-trained model using other data, or extracted from a pre-trained model MN1. For example, model Mn0 can be obtained from pre-trained MN1 by modifying the first two-dimensional convolutional layer L1st of MN1 by retaining only the parameters corresponding to the selected n bands, while the parameters of other layers remain unchanged. In other words, the training samples substantially input into the second spectral processing model only contain spectral responses corresponding to the second band set SBn, without requiring input of spectral responses from bands other than the selected n bands. The second type of input object can be, for example, a labeled spectral image containing spectral responses of n bands (e.g., a hyperspectral image) or an image patch from the aforementioned spectral image. The spectral image corresponding to the second type of input object can be captured by a spectral camera or obtained by extracting the spectral image corresponding to the corresponding n bands from the first type of spectral image.

[0067] In one embodiment, when the number of bands of the input object of model Mn1 (i.e., the "target value") is determined, the band selection method of this disclosure can be performed once to reduce the number of bands of the model from N to the target value in one go (hereinafter referred to as the "one-time pruning method"). Optionally, the number of bands of the model can also be gradually reduced from N to the target value and a model for a band set with the number of bands equal to the target value can be generated by iteratively performing a combination of determining the second band set and training the second spectral image processing model (hereinafter referred to as the "iterative pruning method"). That is, the band selection method of this disclosure can be performed multiple times, and the spectral image processing model can be trained multiple times to generate a spectral image processing model Mn1 with the target number of bands. For example, the number of bands in model M is gradually reduced from 128 to the target value of 16 through the following three iterations to generate model M16: In the first iteration, N = 128, n = 64 is set, the second band set (SBn = SB64) with n = 64 is determined, and the spectral image processing model M64 (Mn = M64) corresponding to this band set is trained. The initial model of model M64 before training can be determined based on model M128 (MN = M128) corresponding to the first band set with 128 bands. That is, in the first iteration, the image processing model M is updated from M128 to the trained M64. In the second iteration, N and n are updated to 64 (i.e., the first band set is updated to the second band set of the previous iteration) and 32, respectively, and the second band set (i.e., n = 32) is determined. The process involves updating the second band set and training a spectral image processing model M32 corresponding to that band set. The initial model of M32 before training can be determined based on model M64 corresponding to the first band set with 64 bands; that is, in the second iteration, model M is updated from M64 to the trained M32. In the third iteration, N and n are updated to 32 (i.e., the first band set is updated to the second band set of the previous iteration) and 16, respectively. A second band set with n = 16 is determined (i.e., the second band set is updated), and a spectral image processing model M16 corresponding to that band set is trained. The initial model of M16 before training can be determined based on model M32 corresponding to the first band set with 32 bands; that is, in the third iteration, model M is updated from M32 to the trained M16. For this example, model M16 is used as the generative spectral image processing model. Iterative pruning methods are more time-consuming and complex than one-time pruning methods, but they select slightly better bands, which results in a slight improvement in the performance of the generated spectral image processing model.

[0068] In one example, when generating a spectral image processing model iteratively, the same number of bands can be reduced in each iteration. In another example, when generating a spectral image processing model iteratively, the same percentage (e.g., 50%) of bands are reduced in each iteration. In yet another example, the number of bands reduced in each iteration can gradually decrease with each iteration. In yet another example, when generating a spectral image processing model iteratively, a small number of previously deleted bands are also selected and reintroduced in each iteration while the number of bands is reduced.

[0069] In one example, when generating a spectral image processing model iteratively, the reduced number of bands n(q) in each iteration can be determined by equation (12).

[0070]

[0071] Where Int is the floor function, P is the final number of bands to be removed, m is the total number of iterations, and q is the current iteration number. The value of q ranges from 0 to m. When q = 0, it means that band selection has not yet started; when q = m, it means that the final band selection is being performed. The aforementioned method of gradually decreasing n(q) is merely an example. In this disclosure, the methods of gradually decreasing n(q) include, but are not limited to, the aforementioned method.

[0072] To verify the advantages of method 100, the inventors conducted some comparative experiments. These experiments were based on histopathological data of cholangiocarcinoma tissue disclosed in the following document 3:

[0073] Document 3: Zhang, Qing, et al., "A multidimensional choledoch database and benchmarks for cholangiocarcinoma diagnosis", IEEE access 7(2019):149414-149421.

[0074] The image data in Reference 3 includes RGB images and hyperspectral images. Only the hyperspectral images were used in the experiment. The dimensions of the hyperspectral images are 1280*1025*60, meaning each image has 60 bands. According to the analysis in Reference 4, the spectral images of the two end bands in the monotonically increasing wavelength band sequence formed by these bands have significant noise:

[0075] Document 4: Yun, Boxiang, et al., "SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation", arXiv preprint arXiv:2103.03604 (2021).

[0076] For the dataset in Reference 3, the inventors divided it into three sets: a training set with 646 images, a validation set with 70 images, and a test set with 173 images. The training model used in the experiment was the commonly used 2D U-net model.

[0077] First, a first-class spectral image processing model was obtained by training the model using objects containing spectral responses corresponding to all 60 bands (the first band set). This model achieved a MIOU (mean intersection-to-union ratio) of 67.17 on the test set. Figure 4 As shown, the L1 norm of band i was calculated as an importance coefficient in the experiment. Figure 4 In this band sequence containing 60 bands, the L1 norm of the bands at both ends is relatively large. Considering the high noise in the band sequences corresponding to the first band set, Figure 4 This indicates that the L1 norm is not very accurate in identifying noise bands. Therefore, for equation (9), a smaller value for θ is preferred.

[0078] In the experiment, the similarity matrix of the first band set was calculated with reference to equation (1). Figure 5 The diagram illustrates the similarity matrix of the first band set in the experiment, where the brightness represents the similarity value, with white indicating a similarity value of 1, meaning the two bands are considered identical. Figure 5 It can be observed that the calculated similarity matrix has a very high similarity between adjacent bands, which has good physical significance.

[0079] The inventors also compared the MIOU of the model under different band selection methods. Figure 6The comparative experimental results are shown. The dashed line parallel to the "n" axis is the reference line, representing the MIOU of the spectral image processing model when n=60, that is, the MIOU of the model using 60 bands without band selection; the polygonal line with rectangles represents the MIOU of the model corresponding to different numbers of bands n when selecting bands using the importance coefficient r (L1 norm) in reference 1; the polygonal line with hollow circles represents the MIOU of the model corresponding to different numbers of bands n when selecting bands using the first band set similarity matrix Mats based on Euclidean distance; the polygonal line with triangles represents the MIOU of the model corresponding to different numbers of bands n when selecting bands using the importance coefficient r and the first band set similarity matrix Mats based on Euclidean distance, as in equation (9), where θ=0.015. It can be seen that the model based on the similarity matrix method has a significant performance improvement over the model based on the importance method, and the model based on both similarity and importance methods has a further performance improvement over the model using only similarity methods.

[0080] This disclosure also provides an apparatus for selecting a band for spectral image processing. See below for reference. Figure 7 An exemplary description is provided. Figure 7 An exemplary block diagram of an apparatus 700 for selecting bands according to an embodiment of the present disclosure is shown. The apparatus 700 includes a generation unit 701 and a selection unit 703. The generation unit 701 is configured to generate a first band set similarity matrix Mats, characterizing the similarity between bands in the N bands, based on a first two-dimensional convolutional layer of a first spectral image processing model trained using a first type of object containing spectral responses corresponding to a first band set of N bands. The selection unit 703 is configured to select n bands from the N bands as a second band set associated with a second spectral image processing model, at least based on the first band set similarity matrix Mats; where N>n; and the second spectral image processing model is configured to perform spectral image processing on a second type of object containing spectral responses corresponding to the second band set. The apparatus 700 corresponds to method 100. Further configuration of the apparatus 700 can be found in the description of method 100 in this disclosure.

[0081] This disclosure also provides an apparatus for generating spectral image processing models. See below for reference. Figure 8 An exemplary description is provided. Figure 8An exemplary block diagram of an apparatus 800 for generating a spectral image processing model according to an embodiment of the present disclosure is shown. The apparatus 800 includes a determination unit 801 and a training unit 803. The determination unit 801 is used to determine a second band set using the band selection method described above in the present disclosure. The training unit 803 is used to train a second spectral image processing model using a second type of objects containing spectral responses corresponding to the second band set to generate a spectral image processing model. The apparatus 800 corresponds to method 300. Further configuration of the apparatus 800 can be found in the description of method 300 in the present disclosure.

[0082] This disclosure also provides an apparatus for selecting a frequency band. See below for reference. Figure 9 An exemplary description is provided. Figure 9 An exemplary block diagram of an apparatus 900 for selecting bands according to an embodiment of the present disclosure is shown. The apparatus 900 includes: a memory 901 storing instructions; and at least one processor 903. The at least one processor 903 is capable of communicating with the memory to execute instructions retrieved from the memory, and the instructions cause the at least one processor to: generate a first band set similarity matrix characterizing the similarity between bands in the N bands based on a first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects containing spectral responses corresponding to a first band set of N bands; and select n bands from the N bands as a second band set associated with a second spectral image processing model, at least based on the first band set similarity matrix; wherein N>n; and the second spectral image processing model is configured to perform spectral image processing on a second class of objects containing spectral responses corresponding to the second band set. The functions of the instructions correspond to those of method 100, and therefore, further details can be found in the description of method 100.

[0083] This disclosure also provides an apparatus for generating a spectral image processing model. The apparatus includes: a memory storing instructions; and at least one processor. The at least one processor is capable of communicating with the memory to execute instructions retrieved from the memory, and the instructions cause the at least one processor to: determine a second band set using the band selection method described above in this disclosure; and train a second spectral image processing model using a second type of object containing spectral responses corresponding to the second band set to generate the spectral image processing model. This apparatus for generating the spectral image processing model corresponds to method 300. Further configuration of the apparatus for generating the spectral image processing model can be found in the description of method 300 in this disclosure.

[0084] One aspect of this disclosure provides a computer-readable storage medium having a program stored thereon, which, when executed, causes a computer to function as: a generation unit for generating a first band set similarity matrix Mats characterizing the similarity between bands in the N bands, based on a first two-dimensional convolutional layer of a first spectral image processing model trained using a first type of object containing spectral responses corresponding to a first band set of N bands; and a selection unit for selecting n bands from the N bands as a second band set associated with a second spectral image processing model, at least based on the first band set similarity matrix Mats; wherein N>n; and the second spectral image processing model for performing spectral image processing on a second type of object containing spectral responses corresponding to the second band set. This program corresponds to the band selection method of this disclosure. Further details regarding this program can be found in the description of the band selection method in this disclosure.

[0085] One aspect of this disclosure provides another computer-readable storage medium having a program stored thereon, which, when executed, causes a computer to function as: a determining unit for determining a second band set using the band selection method described above in this disclosure; and a training unit for training a second spectral image processing model using a second class of objects containing spectral responses corresponding to the second band set to generate a spectral image processing model. This program corresponds to the method for generating a spectral image processing model in this disclosure. Further details regarding this program can be found in the description of the method for generating a spectral image processing model in this disclosure.

[0086] According to one aspect of this disclosure, an information processing device is also provided.

[0087] Figure 10 This is an exemplary block diagram of an information processing device 1000 according to one embodiment of the present disclosure. Figure 10 In this system, the central processing unit (CPU) 1001 performs various processes based on programs stored in the read-only memory (ROM) 1002 or programs loaded from the storage section 1008 into the random access memory (RAM) 1003. The RAM 1003 also stores, as needed, data required by the CPU 1001 when performing various processes.

[0088] CPU 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output interface 1005 is also connected to bus 1004.

[0089] The following components are connected to the input / output interface 1005: an input section 1006 including a soft keyboard, etc.; an output section 1007 including a display such as a liquid crystal display (LCD) and speakers, etc.; a storage section 1008 such as a hard disk; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet, a local area network, a mobile network, or a combination thereof.

[0090] The driver 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a semiconductor memory, is installed on the driver 1010 as needed, so that programs read from it can be installed into the storage section 1008 as needed.

[0091] CPU 1001 can run programs for methods applied to band selection or methods for generating spectral image processing models.

[0092] The scheme disclosed herein includes band selection and generation of a spectral processing model based on a first band set similarity matrix associated with a first two-dimensional convolutional layer of a trained first spectral image processing model. The first band set similarity matrix contains more information than the L1 norm; therefore, band selection methods based on the first band set similarity matrix can perform better than band selection methods based solely on the L1 norm. Considering the first band set similarity matrix, appropriately considering band importance can select a more preferred second band set. The beneficial effects of the methods, apparatus, and storage media of this disclosure include at least one of the following: reducing redundant bands, fewer noisy bands, reducing model complexity, reducing computational costs, improving model accuracy performance, reducing test data acquisition time, and reducing test data acquisition costs.

[0093] As described above, this disclosure provides the principles for selecting bands and generating spectral image processing models. It should be noted that the effects of the solutions in this disclosure are not necessarily limited to those described above, and any effect shown in this specification or other effects that can be understood from this specification can be achieved in addition to or in place of the effects described in the preceding paragraphs.

[0094] Although the invention has been disclosed above through a description of specific embodiments, it should be understood that those skilled in the art can devise various modifications (including, where applicable, combinations or substitutions of features between embodiments), improvements, or equivalents to the invention within the spirit and scope of the appended claims. These modifications, improvements, or equivalents should also be considered to be included within the scope of this disclosure.

[0095] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0096] Furthermore, the methods of the various embodiments of the present invention are not limited to being performed in the chronological order described in the specification or shown in the drawings, but may also be performed in other chronological orders, in parallel, or independently. Therefore, the execution order of the methods described in this specification does not constitute a limitation on the technical scope of the present invention.

[0097] Postscript

[0098] This disclosure includes, but is not limited to, the following schemes.

[0099] 1. A method for selecting a spectral band for spectral image processing, characterized in that it comprises:

[0100] Based on the first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects as training samples, comprising spectral responses corresponding to a first band set of N bands, a first band set similarity matrix is ​​generated, representing the similarity between bands in the N bands; and

[0101] At least based on the first band set similarity matrix, n bands are selected from the N bands as a second band set associated with the second spectral image processing model;

[0102] Where N>n; and

[0103] The second spectral image processing model is configured to perform spectral image processing on a second type of object containing spectral responses corresponding to the second band set.

[0104] 2. The method according to Appendix 1, wherein the n bands are selected based on the determinant of the partial band similarity matrix of some bands among the N bands.

[0105] 3. The method according to Appendix 1, wherein the similarity between bands is determined based on the corresponding weight vector in the weight matrix of the first two-dimensional convolutional layer.

[0106] 4. The method according to Appendix 3, wherein the similarity between each band is determined based on the Euclidean distance or cosine distance between the corresponding weight vectors.

[0107] 5. According to the method described in Appendix 1, selecting n bands from the N bands based at least on the similarity matrix of the first band set includes:

[0108] Based on the similarity matrix of the first band set and the importance coefficient of each band in the N bands, the n bands are selected from the N bands.

[0109] 6. According to the method described in Appendix 5, the importance coefficient of the corresponding band is determined by calculating the Lp norm of the corresponding weight vector in the weight matrix of the first two-dimensional convolutional layer; and

[0110] p≥1.

[0111] 7. The method according to Appendix 5, wherein the n bands are selected based on the weighted sum of the sum of the importance coefficients of some bands among the N bands and the logarithm of the determinant of the partial band similarity matrix of the partial bands.

[0112] 8. A method for generating a spectral image processing model, characterized in that it includes:

[0113] The second band set was determined by using the method described in any one of Appendices 1 to 7; and

[0114] The spectral image processing model is generated by training the second type of object, which contains the spectral response corresponding to the second band set, using the second type of object.

[0115] 9. The method according to Appendix 8, wherein the number of bands in the second band set is gradually reduced to a target value by iteratively performing a combination of determining the second band set and training the second spectral image processing model, and the spectral image processing model for the band set with a number of bands equal to the target value is generated.

[0116] 10. An apparatus for selecting a band for image processing, characterized in that it comprises:

[0117] A memory, on which instructions are stored; and

[0118] At least one processor, the at least one processor being capable of communicating with the memory to execute the instructions fetched from the memory, and the instructions causing the at least one processor to:

[0119] Based on the first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects as training samples, which contains spectral responses corresponding to the first band set of N bands, a first band set similarity matrix representing the similarity between the bands in the N bands is generated; and

[0120] At least based on the first band set similarity matrix, n bands are selected from the N bands as a second band set associated with the second spectral image processing model;

[0121] Where N>n; and

[0122] The second spectral image processing model is configured to perform spectral image processing on a second type of object that contains spectral responses corresponding to the second band set.

[0123] 11. A device for selecting a frequency band, characterized in that it comprises:

[0124] The generation unit is configured to generate a first band set similarity matrix characterizing the similarity between bands in the N bands, based on the first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects as training samples, comprising spectral responses corresponding to a first band set of N bands; and

[0125] The selection unit is configured to select n bands from the N bands as a second band set associated with the second spectral image processing model, based at least on the first band set similarity matrix.

[0126] Where N>n; and

[0127] The second spectral image processing model is configured to perform spectral image processing on a second type of object that contains spectral responses corresponding to the second band set.

[0128] 12. The apparatus according to Appendix 11, wherein the selection unit is configured to select the n bands based on the determinant of a partial band similarity matrix of partial bands among the N bands.

[0129] 13. The apparatus according to Appendix 11, wherein the generation unit is configured to determine the similarity between bands based on the corresponding weight vector in the weight matrix of the first two-dimensional convolutional layer.

[0130] 14. The apparatus according to Appendix 13, wherein the generating unit is configured to determine the similarity between bands based on the Euclidean or cosine distance between the corresponding weight vectors.

[0131] 15. The apparatus according to Appendix 11, selecting n bands from the N bands based at least on the similarity matrix of the first band set, comprises:

[0132] Based on the similarity matrix of the first band set and the importance coefficient of each band in the N bands, the n bands are selected from the N bands.

[0133] 16. The apparatus according to Appendix 15, wherein the importance coefficient of the corresponding band is determined by calculating the Lp norm of the corresponding weight vector in the weight matrix of the first two-dimensional convolutional layer; and

[0134] p≥1.

[0135] 17. The apparatus according to Appendix 15, wherein the n bands are selected based on a weighted sum of the sum of importance coefficients of some bands among the N bands and the logarithm of the determinant of the partial band similarity matrix of the partial bands.

[0136] 18. The apparatus according to Appendix 17, wherein the weighting factor of the sum is less than 0.2.

[0137] 19. The apparatus according to Appendix 12, wherein the selection unit is configured to solve for the selected n bands by a greedy algorithm.

[0138] 20. The apparatus according to Appendix 19, wherein the first type of object is a hyperspectral image containing the spectral response corresponding to the first band set of the N bands.

Claims

1. A method for selecting a spectral band for spectral image processing, characterized in that, include: Based on the weight matrix of the first two-dimensional convolutional layer of a first spectral image processing model trained using a first class of objects as training samples containing spectral responses corresponding to a first band set of N bands, a first band set similarity matrix is ​​generated to characterize the similarity between the bands in the N bands. as well as At least based on the first band set similarity matrix, n bands are selected from the N bands as a second band set associated with the second spectral image processing model; Where N>n; and The second spectral image processing model is configured to perform spectral image processing on a second type of object that contains spectral responses corresponding to the second band set.

2. The method according to claim 1, wherein, The n bands are selected based on the determinant of the partial band similarity matrix of some bands among the N bands.

3. The method according to claim 1, wherein, The similarity between each band is determined based on the corresponding weight vector in the weight matrix of the first two-dimensional convolutional layer.

4. The method according to claim 3, wherein, The similarity between each band is determined based on the Euclidean or cosine distance between the corresponding weight vectors.

5. The method according to claim 1, wherein selecting n bands from the N bands based at least on the similarity matrix of the first band set comprises: Based on the similarity matrix of the first band set and the importance coefficient of each band in the N bands, the n bands are selected from the N bands.

6. The method according to claim 5, wherein, The importance coefficient of the corresponding band is determined by calculating the Lp norm of the weight vector in the weight matrix of the first two-dimensional convolutional layer; and p≥1。 7. The method according to claim 5, wherein, The n bands are selected based on the weighted sum of the importance coefficients of some bands among the N bands and the logarithm of the determinant of the partial band similarity matrix of the partial bands.

8. A method for generating a spectral image processing model, characterized in that, include: The second band set is determined by using the method of any one of claims 1 to 7; and The spectral image processing model is generated by training the second type of object, which contains the spectral response corresponding to the second band set, using the second type of object.

9. The method according to claim 8, wherein, The number of bands in the second band set is gradually reduced to a target value by iteratively performing a combination of operations to determine the second band set and train the second spectral image processing model, and the spectral image processing model for the band set with a number of bands equal to the target value is generated.

10. An apparatus for selecting a wavelength band for spectral image processing, characterized in that, include: A memory that stores instructions; as well as At least one processor, the at least one processor being capable of communicating with the memory to execute the instructions fetched from the memory, and the instructions causing the at least one processor to: Based on the weight matrix of the first two-dimensional convolutional layer of the first spectral image processing model trained using a first class of objects containing spectral responses corresponding to the first band set of N bands as training samples, a first band set similarity matrix is ​​generated to characterize the similarity between the bands in the N bands. as well as At least based on the first band set similarity matrix, n bands are selected from the N bands as a second band set associated with the second spectral image processing model; Where N>n; and The second spectral image processing model is configured to perform spectral image processing on a second type of object that contains spectral responses corresponding to the second band set.

Images