Endoscope multispectral image processing system and processing and training method

Abstract

The present application relates to medical technology, and discloses an endoscopic multispectral image processing system and a processing and training method, which can select wavelengths based on deep learning feature sorting, so that doctors can see the lesions more easily. The system includes: an encoder, configured to encode n endoscopic images obtained under n input wavelengths respectively, and output n feature maps corresponding to n different wavelengths, where n is an integer greater than 1; a wavelength selection module, configured to output a weight matrix according to the n input feature maps, the weight matrix represents the importance of feature maps of different wavelengths, and the weight matrix is ​​used to control the feature map of the input splicing module; the splicing module is configured to splice the feature map output by the encoder to obtain a spliced ​​feature map; a classifier is configured to classify the spliced ​​feature map and output the classification result.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to an endoscopic multispectral image processing technology. Background Art

[0002] With the development of modern science and technology, the application of medical endoscopes in clinical practice has become more and more popular, and has become one of the important tools for humans to observe and diagnose organ lesions in the human body. Medical endoscopes are currently widely used medical devices. Medical endoscopes are an effective, non-invasive and low-cost medical diagnostic tool that can display images of diseased tissues in the human body. Studies have shown that as the gold standard for screening of gastrointestinal tumors, gastrointestinal endoscopy can reduce the risk of death from gastrointestinal tumors by detecting tumors at an early stage and removing precancerous lesions.

[0003] Ordinary electronic endoscopes use white light as the illumination source. Although they can provide clear images of digestive tract mucosal tissue and detect lesions with obvious changes in morphology or color, it is difficult to observe small and flat early cancers and blood vessels at all levels in the digestive tract, and it is also impossible to improve the contrast of fine structures, which is prone to missed diagnosis and misdiagnosis. In order to overcome these shortcomings, endoscopes using multispectral imaging technology came into being. They mainly include the narrow band imaging technology NBI (Narrow Band Imaging) developed by Olympus, the autofluorescence imaging technology AFI (Auto Fluorescence Imaging), the new image enhancement I-Scan technology invented by Japan's Pentax, the intelligent spectroscopic colorimetric technology FICE (Fuji Intelligent Chromo Endoscopy) developed by Japan's Fujinon, and the blue laser imaging technology BLI (Blue Laser Imaging) developed by Japan's Fujifilm.

[0004] Although different studies have suggested that narrow-band light of certain wavelengths can improve the contrast between lesions and surrounding tissues in clinical detection, for example, the NBI imaging technology designed by Olympus Corporation of Japan selected narrow-band light of 415nm and 540nm, and the blue laser imaging technology BLI (Blue Laser Imaging) developed by Fujifilm Corporation of Japan uses semiconductor lasers as light sources, in which the narrow-band light mode includes two wavelengths of 410nm and 450nm. Other studies have shown that the 550-700nm band has also been shown to improve the clinical images of some diseases. However, there is no unified consensus on which specific wavelengths of narrow-band light have the best effect on image improvement, and the choice of the intensity of each wavelength of narrow-band light.

[0005] Moreover, although the multispectral system covers most of the narrowband light in the visible light band, narrowband light of some wavelengths may not be significantly helpful for the prediction results of the neural network or the doctor's observation. In addition, narrowband light of some wavelengths (such as 415nm and 540nm selected by NBI technology) may be more helpful for the prediction results and the doctor's observation. This may be because the features contained in the pictures under different wavelengths of light may be of different importance. For example, NBI imaging technology uses narrowband light of 415nm and 540nm wavelengths, because the larger blood vessels under the mucosa can absorb blue light and appear blue-green, while the microvessels can absorb blue-green and appear brown, thereby improving the contrast between the microvessels and the surrounding tissues. Therefore, NBI technology believes that the features of the pictures under the two wavelengths of 415nm and 540nm are more important and more conducive to the doctor's observation.

[0006] Therefore, how to automatically select the appropriate wavelength so that doctors can more easily see the lesions is a very important issue in this field. Summary of the invention

[0007] The purpose of this application is to provide an endoscopic multispectral image processing system and a training and processing method, so that doctors can see the lesions more clearly.

[0008] The present application discloses an endoscope multispectral image processing system, comprising:

[0009] An encoder is configured to encode n endoscopic images obtained at n input wavelengths respectively, and output n feature maps corresponding to n different wavelengths, where n is an integer greater than 1;

[0010] The wavelength selection module is configured to output a weight matrix according to the n input feature maps, wherein the weight matrix represents the importance of feature maps of different wavelengths, and the weight matrix is ​​used to control the feature map input to the splicing module;

[0011] The splicing module is configured to splice the feature maps output by the encoder to obtain a spliced ​​feature map;

[0012] The classifier is configured to classify the concatenated feature map and output a classification result.

[0013] In a preferred example, it also includes a feature map processing module, which is configured to screen and / or weight the n feature maps output by the encoder according to the weight matrix and then output them to the splicing module.

[0014] In a preferred embodiment,

[0015] The weight matrix includes n weights, corresponding to characteristic graphs of n different wavelengths respectively;

[0016] The screening includes screening out the characteristic graph corresponding to the wavelength corresponding to the weight less than a preset threshold value among the n weights of the weight matrix;

[0017] The weighted processing includes multiplying the characteristic maps by corresponding weights output by the wavelength selection module.

[0018] In a preferred example, the weight matrix is ​​used to control the feature map input to the splicing module, further comprising:

[0019] The preprocessing module is configured to screen and / or weight the n endoscopic images obtained at n wavelengths input to the encoder according to the weight matrix.

[0020] In a preferred embodiment, it also includes an imaging device for capturing endoscopic images;

[0021] The imaging device is configured to control an imaging wavelength used or received for capturing an endoscopic image according to the weight matrix.

[0022] In a preferred embodiment,

[0023] The control includes screening and / or light intensity manipulation;

[0024] The weight matrix includes n weights, corresponding to characteristic graphs of n different wavelengths respectively;

[0025] The screening includes screening out light of wavelengths corresponding to weights less than a preset threshold among the n weights;

[0026] The light intensity processing includes controlling the intensity of light of each wavelength according to the n weights.

[0027] In a preferred example, the system uses multiple sets of training data for supervised training, each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images.

[0028] In a preferred example, there are n encoders, each of which is configured to encode an endoscopic image obtained at an input wavelength to obtain a feature map corresponding to the wavelength. A total of n endoscopic images obtained at different wavelengths are input into the n encoders respectively, and n feature maps corresponding to the n different wavelengths are output.

[0029] In a preferred example, the n encoders are n deep neural networks; the classifier is a deep neural network; and the wavelength selection module is a deep neural network based on self-attention.

[0030] The present application also discloses an endoscopic multispectral image processing method comprising:

[0031] Acquire n endoscopic images obtained at n different wavelengths;

[0032] Inputting n endoscopic images obtained at n different wavelengths into the encoder of the endoscopic multispectral image processing system as described above, respectively, to obtain n weights corresponding to the n different wavelengths output by the wavelength selection module of the system;

[0033] Selecting a weight greater than a preset threshold from the n weights;

[0034] Output an endoscopic image corresponding to the wavelength corresponding to the weight greater than the preset threshold.

[0035] The present application also discloses a training method for an endoscope multispectral image processing system, including:

[0036] Preparing multiple sets of training data, each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images;

[0037] The endoscopic multispectral image processing system as described above is trained as a whole using the multiple sets of data, wherein for each set of training data, the n endoscopic images are used as inputs of the encoder of the system, and the classification results are used as outputs of the classifier of the system.

[0038] The present application also discloses an endoscopic multispectral image processing method comprising:

[0039] Inputting n endoscopic images obtained at n different wavelengths into n encoders of the endoscopic multispectral image processing system described above, respectively;

[0040] The output result of the classifier of the endoscope multispectral image processing system is obtained and displayed on a display.

[0041] In the implementation mode of the present application, the neural network in the system automatically ranks the importance of each wavelength of light, and then feeds back to the system. The system automatically selects the appropriate wavelength and light intensity for re-imaging, and inputs the new image into the classification network in the next step for pathological classification, and adjusts and outputs the new image for the doctor to observe, so that the doctor can easily see the lesion. Specifically,

[0042] 1) Based on the self-attention model, the neural network automatically ranks the importance of each wavelength of light.

[0043] 2) Let the system automatically learn the importance of different spectral images, including: normalizing to the range of 0 to 1 through a Softmax or Sigmoid activation function, generating a weight matrix W containing n elements. The weight coefficients w1 to wn in the weight matrix represent the importance of the image under wavelengths L1 to Ln. The larger the weight, the greater the contribution of the wavelength to the system prediction; the smaller the weight, the smaller the contribution of the wavelength to the system prediction.

[0044] 3) Before splicing, find the wavelength corresponding to the weight less than the preset threshold among the n weights and set the feature map corresponding to the corresponding wavelength to 0. Because all the n wavelengths may contain noise wavelengths, although the noise wavelengths can be filtered out to a certain extent by the above-mentioned self-attention wavelength selection module, it is not as direct as deleting the noise. Therefore, the system of the present application sets all the images corresponding to the filtered wavelengths to black. The input of the neural network of the system of the present application is the image corresponding to the wavelength after being screened by the above-mentioned network, that is, the image corresponding to the wavelength greater than the threshold is retained, and the image corresponding to the wavelength less than the threshold is set to black, and the classification result is output. Similarly, during training, supervised training is performed, and each set of training data includes m images at m wavelengths greater than the threshold and nm black images less than the threshold, as well as pre-calibrated classification results.

[0045] 4) The image selection module adjusts the light source according to the weights corresponding to the n wavelengths. i The light intensity of wavelengths less than the threshold t is set to 0, and the light intensity ratio of wavelengths with weight wi greater than the threshold t is the same as the weight ratio. In this way, the noise wavelength is eliminated, making the image clearer and improving the shooting efficiency.

[0046] The specification of the present application records a large number of technical features, which are distributed in various technical solutions. If all possible combinations of technical features of the present application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of the present application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions should all be regarded as having been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be regarded as having been recorded because it is technically infeasible, and the solution of A+B+C+E should be regarded as having been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic diagram of the structure of an endoscope multispectral image processing system according to the present application;

[0048] Figure 2 is a structural schematic diagram of an endoscope multispectral image processing system according to an embodiment of the present application;

[0049] Figure 3 This is a schematic diagram of a structure of obtaining new images and classifying them according to an embodiment of an endoscope multispectral image processing system of the present application by adjusting the wavelength of a light source according to a feature matrix;

[0050] Figure 4 It is a structural schematic diagram of a wavelength selection module based on CNN+XGBoost according to the endoscope multispectral image processing system of the present application;

[0051] Figure 5 It is a structural schematic diagram of a classifier according to an embodiment of a wavelength selection module based on CNN+XGBoost of an endoscope multispectral image processing system of the present application;

[0052] Figure 6 is a schematic structural diagram of an OOB-based wavelength selection module of an endoscope multispectral image processing system according to the present application;

[0053] Figure 7 is a flow chart of an endoscopic multispectral image processing method according to an embodiment of the present application;

[0054] Figure 8 is a schematic diagram of the use process of the endoscope multispectral image processing system according to the present application;

[0055] Fig. 9 It is a schematic diagram of a real-time target detection model of polyps in an endoscopic multispectral image processing system according to the present application. DETAILED DESCRIPTION

[0056] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0058] The first embodiment of the present application relates to an endoscopic multispectral image processing system, such as Figure 1 As shown, the endoscope multispectral image processing system includes: an encoder, a wavelength selection module, a splicing module and a classifier.

[0059] The encoder is configured to encode n endoscopic images obtained at n input wavelengths respectively, and output n feature maps corresponding to n different wavelengths, where n is an integer greater than 1. There are n encoders, each of which is configured to encode an endoscopic image obtained at one input wavelength to obtain a feature map corresponding to the wavelength. A total of n endoscopic images obtained at different wavelengths are input into n encoders respectively, and n feature maps corresponding to n different wavelengths are output.

[0060] Optionally, in one embodiment, the system of the present application is used to collect images of the same area under the illumination of narrow-band light of different wavelengths, and the dimension of the spectral wavelength is added on the basis of the two-dimensional image to form a data cube or hypercube. The multi-spectrum includes visible light with a wavelength between 400nm and 700nm. Assuming that a central wavelength is taken every 10nm for imaging, 31 images of the same area under different wavelengths of light can be obtained, and then a multi-spectral image with a spectral dimension of 31 can be obtained. In this embodiment, n=31. In other embodiments, n can also be other values, such as 5, 15, etc.

[0061] The images L1~Ln at each wavelength in the data cube are input into their respective encoders 1~n for feature extraction. These encoders are composed of deep neural networks, including convolutional layers, downsampling layers, pooling layers, batch normalization layers, etc. These encoders can share all or part of the parameters, and obtain their corresponding features after passing through the encoder. Figure 1 ~n.

[0062] The wavelength selection module is configured to output a weight matrix based on the n input feature maps. The weight matrix represents the importance of the feature maps of different wavelengths. The larger the weight, the greater the contribution of the wavelength to the network prediction; the smaller the weight, the smaller the contribution of the wavelength to the system prediction. The weight matrix is ​​used to control the feature map of the input splicing module.

[0063] The wavelength selection module can be implemented in many ways.

[0064] Optionally, in one embodiment, Figure 2As shown in the figure, a wavelength selection module is implemented using a neural network based on a self-attention mechanism. After the multispectral image is input into the wavelength selection module, the wavelength selection module concatenates these feature maps in the channel dimension. The concatenated feature maps pass through several fully connected layers and activation layers, and finally are normalized to the range of 0 to 1 through a Softmax or Sigmoid activation function to generate a weight matrix W containing n elements. The weight coefficients w1~wn in the weight matrix represent the importance of the image under the wavelength L1~Ln, and w1~wn are all between 0 and 1.

[0065] Optionally, in another embodiment, the wavelength selection module obtains a weight matrix based on CNN+XGBoost, such as Figure 4 shown.

[0066] 1) First, a deep neural network model is used to extract feature vectors. The network includes an encoder, a global average pooling layer (GAP), and a classifier. Supervised training is used. Each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images. The image at each wavelength is passed through encoders 1 to n to obtain a w*h*c feature. Figure 1 ~n, w, h are the width and height of the feature map, and c is the number of feature map channels. After the feature maps are spliced ​​in the channel dimension, they pass through the global pooling layer, and each feature map obtains a 1*c feature vector. That is, each image under each wavelength corresponds to a 1*c feature vector, and the total length of the feature vector is n*c, where n is the number of wavelengths.

[0067] 2) If Figure 5 As shown, an XGBoost model is established, and the feature vector obtained by the above-trained deep neural network model is used as the input of the XGBoost model. Supervised training is used, and each set of training data includes feature vectors obtained by the above-mentioned deep neural network model for n endoscopic images obtained at n different wavelengths and classification results pre-calibrated based on the n endoscopic images.

[0068] 3) Use the XGBoost model to calculate the importance score of the input feature. For the input n*c feature vector, the importance score matrix S = [s1,…,s c ,s c+1 ,…,s 2*c ,…,s (n-1)*c+1 ,…,s n*c ], for wavelength L i The importance score of the corresponding image [s (i-1)*c+1 ,…,s i*c] and sum them to get the feature weight wi corresponding to the image at wavelength Li. Calculate the feature weights w1~wn corresponding to the images at wavelengths L1~Ln in sequence, and then normalize w1~wn to [0,1] through the sigmoid / softmax function to get the weight matrix W=[w1,…,wn].

[0069] Optionally, in another embodiment, the wavelength selection module sorts the wavelengths based on OOB, and the sorting training process and steps further include:

[0070] Train the neural network model, then use OOB (Out of Bag) data to calculate feature importance. After the model is trained, use Out of Bag (test) data to quantify feature importance. Specifically, first use the trained model to score the OOB data, calculate the accuracy or AUC (Area Under Curve) or other defined evaluation indicators; then perform the following operations on the images at each wavelength in the OOB data:

[0071] 1) If Figure 6 As shown, a classifier model is trained, which can be based on a deep neural network or a traditional machine learning method (such as SVM, XGBoost, etc.). Using supervised training, each set of training data includes n endoscopic images obtained at n different wavelengths and classification results pre-calibrated based on the n endoscopic images;

[0072] 2) Use the trained model to score the test data and calculate the accuracy or AUC (Area Under Curve) or other defined evaluation indicators;

[0073] 3) Set all images at a certain wavelength in the test set to random noise, and keep the images at other wavelengths unchanged, for example, make the noise image satisfy Gaussian distribution; re-score the current test data set and calculate the evaluation index;

[0074] 4) Calculate the index change rate △, which is the difference between the evaluation index of the original test set data and the evaluation index after the image at a certain wavelength is set as random noise. The larger the difference, the more the model performance decreases after removing the wavelength, and the greater the impact of the wavelength on the prediction result; the smaller the difference, the model performance decreases slightly after removing the wavelength, and the smaller the impact of the wavelength on the prediction result.

[0075] 5) Set the images under the wavelengths L1 to Ln in the test set as random noise in turn. Obtain n change rates △1 to △ n , and then use the sigmoid / softmax function to convert △1 to △ nNormalize to [0, 1] to obtain the weight matrix W = [w1, …, wn].

[0076] There are various implementation ways to control the feature maps of the input splicing module using the weight matrix.

[0077] Optionally, in one embodiment, as Figure 3 shown, it further includes a feature map processing module, configured to screen and / or weight the n feature maps output by the encoder according to the weight matrix and then output them to the splicing module.

[0078] The weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths.

[0079] Screening includes screening out the feature maps corresponding to the wavelengths of the weights less than the preset threshold among the n weights of the weight matrix. Set a preset threshold t. If wi < t, then it is considered that the spectral image Li corresponding to the wavelength of this wi has no significance for network prediction. Then the wavelengths (wi < t) that have no significance for network prediction can be screened out through this weight matrix W.

[0080] Weighting processing includes multiplying the feature maps by the corresponding weights output by the wavelength selection module respectively. The pictures L1~Ln at each wavelength in the data cube are respectively input into their respective encoders 1~n for feature extraction, and then the features Figure 1 ~n are respectively multiplied by the weight coefficients w1~wn, and feature fusion is achieved by splicing in the channel dimension, and then sent to the classifier for classification to obtain the final pathological classification result. In one embodiment, by introducing the above screening process, since the number of wavelengths after screening changes from n to m, and there are no corresponding pictures for some wavelengths L i , all the input pixels corresponding to the screened wavelengths are set to (0, 0, 0), that is, black.

[0081] Optionally, these encoders are composed of deep neural networks, including convolutional layers, downsampling layers, pooling layers, batch normalization layers, etc. These encoders can share all or part of the parameters, and the respective corresponding features Figure 1 ~n are obtained after passing through the encoders.

[0082] Optionally, in another embodiment, it further includes a preprocessing module, configured to screen and / or weight the n endoscopic images obtained at n wavelengths input to the encoder according to the weight matrix.

[0083] The weight matrix includes n weights, respectively corresponding to the feature maps of n different wavelengths.

[0084] Screening includes screening out the images corresponding to the wavelengths of the weights less than the preset threshold among the n weights, or setting the light intensity corresponding to the wavelengths with weights less than the threshold to zero in the received endoscopic images.

[0085] The weighted processing includes controlling the intensity of each wavelength of light according to n weights. After receiving n endoscopic images, the intensity of light of different wavelengths in the image is adjusted according to the weights. Optionally, the light intensity I1:I2:…:In=w1:w2:…:wn.

[0086] Optionally, in another embodiment, it also includes an imaging device for capturing endoscopic images; the imaging device is configured to control the imaging wavelength used or received for capturing the endoscopic image according to a weight matrix, including screening and / or light intensity processing to obtain a new multispectral image, which is then re-input into the encoder, subjected to weighted processing, and then input into the stitching module.

[0087] The weight matrix includes n weights, corresponding to characteristic graphs of n different wavelengths.

[0088] The screening includes screening out the light of wavelengths corresponding to the weights less than a preset threshold value among the n weights. Optionally, the light source does not use the light of wavelengths with such small weights for shooting, or the image receiving device does not accept the light of wavelengths with such small weights.

[0089] Light intensity processing includes controlling the intensity of each wavelength of light according to n weights. Optionally, the light source determines the intensity of each wavelength of light emitted by the ratio of wi, and the light intensity I1:I2:…:In=w1:w2:…:wn; or the image receiving device determines the intensity of each wavelength of light to be received by the ratio of wi, and the light intensity I1:I2:…:In=w1:w2:…:wn.

[0090] The splicing module is configured to splice the feature maps output by the encoder to obtain a spliced ​​feature map. These feature maps are spliced ​​in the channel dimension, and the spliced ​​feature maps are sent to the classifier to obtain the final pathology classification result.

[0091] The classifier is configured to classify the concatenated feature map and output a classification result. Optionally, in one embodiment, the classifier can be composed of a deep neural network, including a convolution layer, a downsampling layer, a pooling layer, a batch normalization layer, a fully connected layer, a Softmax / Sigmoid activation layer, etc.

[0092] Optionally, in one embodiment, the n encoders are n deep neural networks; the classifier is a deep neural network; and the wavelength selection module is a deep neural network based on self-attention.

[0093] The system uses multiple sets of training data for supervised training, each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images.

[0094] The second embodiment of the present application relates to an endoscopic multispectral image processing method, such as Figure 7 As shown, including:

[0095] In step 701, n endoscopic images obtained at n different wavelengths are acquired.

[0096] There are many ways to acquire n endoscopic images.

[0097] Optionally, in one embodiment, step 701 further includes switching n types of illumination lights, acquiring n endoscopic images under each type of illumination light, each type of illumination light having a different central wavelength, and a wavelength range of each type of illumination light being smaller than a predetermined threshold.

[0098] Optionally, in another embodiment, step 701 further includes switching the filter device in front of the camera used to capture endoscopic images n times to obtain n endoscopic images, and after each switching of the filter device, only light in a predetermined wavelength range can enter the camera, and after each switching of the filter device, the wavelength range that can enter the camera is different.

[0099] Then, the process proceeds to step 702, and the n endoscopic images obtained at n different wavelengths are respectively input into the encoder of the endoscopic multispectral image processing system of the first embodiment to obtain n weights corresponding to the n different wavelengths output by the wavelength selection module of the system.

[0100] Then proceed to step 703 to select a weight greater than a preset threshold from the n weights.

[0101] Then, the process proceeds to step 704 to output the endoscopic image corresponding to the wavelength corresponding to the weight greater than the preset threshold.

[0102] The image input to the encoder can come from a video, and the wavelength corresponding to the image in the video is switched in sequence under n different wavelengths, and one of each of the n consecutive endoscopic images is selected for output, and the above steps are continuously performed, so that the output endoscopic images also constitute a continuous video. The output image or video can be displayed on a monitor.

[0103] It should be noted that this embodiment is an image processing method embodiment that is compatible with the first embodiment. The technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment.

[0104] A third embodiment of the present application relates to a training method for an endoscope multispectral image processing system, comprising:

[0105] A plurality of sets of training data are prepared, each set of training data including n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images.

[0106] The endoscopic multispectral image processing system as in the first embodiment is trained as a whole using multiple sets of data, wherein for each set of training data, n endoscopic images are used as inputs to the system's encoder, and the classification results are used as outputs of the system's classifier.

[0107] It should be noted that this embodiment is a training method implementation method that is compatible with the first embodiment. The technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment.

[0108] A fourth embodiment of the present application relates to an endoscopic multispectral image processing method, comprising:

[0109] The n endoscopic images obtained at n different wavelengths are respectively input into n encoders of the endoscopic multispectral image processing system of the first embodiment.

[0110] Acquire and display on the display the output results of the classifier of the endoscopic multispectral image processing system.

[0111] It should be noted that this embodiment is an image processing method embodiment that is compatible with the first embodiment. The technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment.

[0112] In order to better understand the technical solution of the present application, a specific example is provided below for illustration. The details listed in the example are mainly for ease of understanding and are not intended to limit the scope of protection of the present application.

[0113] like Figure 8 As shown, the CAD system is the system described in the first embodiment of the present application, and the system mainly includes the following steps:

[0114] 1) Doctors use white light mode for polyp screening, and the system detects the location of polyps in real time. Fig. 9 This is a real-time polyp target detection model. The image x under white light is used as input, and an encoder is input for feature extraction. The feature map obtained after feature extraction is then regressed through a detector to obtain the coordinate y of the polyp location. The encoder and detector are composed of a deep neural network, including convolutional layers, downsampling layers, upsampling layers, pooling layers, batch normalization layers, activation layers, etc.

[0115] 2) When a polyp is found, the system will indicate to the doctor the area where the polyp is located with a red frame on the monitor and sound an alarm. The image processing center will feed back to the light source, switch to multispectral mode, and obtain multispectral images of the same area under continuous narrow-band light of multiple wavelengths.

[0116] 3) The system analyzes multispectral images in real time and gives a judgment on the nature of the pathology by establishing a good spectral feature sorting and interpretability model. At the same time, according to this model, the most appropriate single or multiple wavelength narrowband light and the intensity of each wavelength light are automatically selected, and the light source is fed back to the image processing center for switching.

[0117] 4) The system re-judges the nature of the pathology through the new multispectral image and gives the final judgment of the pathology based on the doctor's judgment based on the new image.

[0118] It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the application documents of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0119] This specification includes combinations of the various embodiments described herein. Individual references to "one embodiment" or a particular embodiment, etc., do not necessarily refer to the same embodiment; however, these embodiments are not mutually exclusive unless indicated as mutually exclusive or clear to those skilled in the art. It should be noted that the word "or" is used in this specification in a non-exclusive sense unless the context clearly indicates or requires otherwise.

[0120] All documents mentioned in this application are considered to be included in the disclosure of this application as a whole, so that they can be used as the basis for modification when necessary. In addition, it should be understood that after reading the above disclosure of this application, those skilled in the art can make various changes or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. An endoscope multispectral image processing system, characterized in that: include: An encoder is configured to encode n endoscopic images obtained at n input wavelengths respectively, and output n feature maps corresponding to n different wavelengths, where n is an integer greater than 1; The wavelength selection module is configured to output a weight matrix according to the n input feature maps, wherein the weight matrix represents the importance of feature maps of different wavelengths, and the weight matrix is ​​used to control the feature map of the input splicing module, wherein the method for obtaining the weight matrix is ​​as follows: The image at n wavelengths is passed through the encoder to obtain n w*h*c feature maps, where w and h are the width and height of the feature map, and c is the number of channels of the feature map; After the feature map is spliced ​​in the channel dimension, it passes through the global pooling layer to obtain a 1*c feature vector corresponding to the image at each wavelength, and the total length of the feature vector is n*c, where n is the number of wavelengths; The feature vector of length n*c is used as the input of the XGBoost model to obtain the importance score matrix S=[s1,…, s c , s c+1 , …, s 2*c , …, s (n-1)*c+1 , …, s n*c ], for wavelength L i The importance score of the corresponding image [s (i-1)*c+1 , …, s i*c ] sum them to obtain the feature weight wi corresponding to the image at wavelength Li, calculate the feature weights w1~wn corresponding to the image at wavelengths L1~Ln in turn, and then normalize w1~wn to [0, 1] through the sigmoid / softmax function to obtain the weight matrix; The splicing module is configured to splice the feature maps output by the encoder to obtain a spliced ​​feature map; The classifier is configured to classify the concatenated feature map and output a classification result.

2. The endoscopic multispectral image processing system according to claim 1, characterized in that: It also includes a feature map processing module, which is configured to screen and / or weight the n feature maps output by the encoder according to the weight matrix and then output them to the splicing module.

3. The endoscopic multispectral image processing system according to claim 2, characterized in that: The weight matrix includes n weights, corresponding to characteristic graphs of n different wavelengths respectively; The screening includes screening out the characteristic graph corresponding to the wavelength corresponding to the weight less than a preset threshold value among the n weights of the weight matrix; The weighted processing includes multiplying the characteristic maps by corresponding weights output by the wavelength selection module.

4. The endoscopic multispectral image processing system according to claim 1, characterized in that: It also includes a pre-processing module configured to screen and / or weight the n endoscopic images obtained at n wavelengths input to the encoder according to the weight matrix.

5. The endoscopic multispectral image processing system according to claim 1, characterized in that: Also included is an imaging device for capturing endoscopic images; The imaging device is configured to control an imaging wavelength used or received for capturing an endoscopic image according to the weight matrix.

6. The endoscopic multispectral image processing system according to claim 5, characterized in that: The control includes screening and / or light intensity manipulation; The weight matrix includes n weights, corresponding to characteristic graphs of n different wavelengths respectively; The screening includes screening out light of wavelengths corresponding to weights less than a preset threshold among the n weights; The light intensity processing includes controlling the intensity of light of each wavelength according to the n weights.

7. The endoscopic multispectral image processing system according to claim 1, characterized in that: The system uses multiple sets of training data for supervised training, each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images.

8. The endoscopic multispectral image processing system according to claim 1, characterized in that: There are n encoders, each of which is configured to encode an endoscopic image obtained at an input wavelength to obtain a feature map corresponding to the wavelength. A total of n endoscopic images obtained at different wavelengths are input into the n encoders respectively, and n feature maps corresponding to the n different wavelengths are output.

9. The endoscopic multispectral image processing system according to claim 8, characterized in that: The n encoders are n deep neural networks; the classifier is a deep neural network; and the wavelength selection module is a deep neural network based on self-attention.

10. An endoscopic multispectral image processing method, characterized in that: include: Acquire n endoscopic images obtained at n different wavelengths; Inputting n endoscopic images obtained at n different wavelengths into the encoder of the endoscopic multispectral image processing system as claimed in claim 1 respectively, the images at n wavelengths are passed through the encoder to obtain n w*h*c feature maps, where w and h are the width and height of the feature map, and c is the number of channels of the feature map; After the feature map is spliced ​​in the channel dimension, it passes through the global pooling layer to obtain a 1*c feature vector corresponding to the image at each wavelength, and the total length of the feature vector is n*c, where n is the number of wavelengths; The feature vector of length n*c is used as the input of the XGBoost model to obtain the importance score matrix S=[s1,…, s c , s c+1 , …, s 2*c , …, s (n-1)*c+1 , …, s n*c ], for wavelength L i The importance score of the corresponding image [s (i-1)*c+1 , …, s i*c ] sum them to obtain the feature weight wi corresponding to the image at wavelength Li, calculate the feature weights w1~wn corresponding to the image at wavelengths L1~Ln in turn, and then normalize w1~wn to [0, 1] through the sigmoid / softmax function to obtain the n weights corresponding to the n different wavelengths output by the wavelength selection module of the system; select a weight greater than a preset threshold from the n weights; Output an endoscopic image corresponding to the wavelength corresponding to the weight greater than the preset threshold.

11. A training method for an endoscope multispectral image processing system, characterized in that: include: Preparing multiple sets of training data, each set of training data includes n endoscopic images obtained at n different wavelengths and pre-calibrated classification results based on the n endoscopic images; The endoscopic multispectral image processing system as described in claim 1 is trained as a whole using the multiple sets of data, wherein, for each set of training data, the n endoscopic images are used as input to the encoder of the system, and the classification result is used as the classifier output of the system.

12. An endoscopic multispectral image processing method, characterized in that: include: inputting n endoscopic images obtained at n different wavelengths into the encoder of the endoscopic multispectral image processing system as claimed in claim 1 respectively; The output result of the classifier of the endoscope multispectral image processing system is obtained and displayed on a display.

Images