A method and system for improving target recognition accuracy in an endoscope NBI mode

Abstract

The present application relates to the field of target recognition under endoscope NBI mode, and particularly to a method for improving target recognition accuracy under endoscope NBI mode, which comprises the following steps: under HSV color space, calculating the mean and mean square deviation of each channel of the image under NBI mode; converting the image under non-NBI mode into HSV color space, normalizing the pixel value of the image under non-NBI mode to the distribution of the mean and mean square deviation, obtaining the first image under non-NBI mode, i.e. making the image under non-NBI mode have similar characteristics of the image under NBI mode, and then equivalent to having a large number of images under NBI mode; putting the image under NBI mode and the first image under non-NBI mode into a preset model for training, obtaining a target model, and thus significantly improving the recognition accuracy of the target under NBI mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target recognition under endoscope NBI mode, in particular to a method and system for improving target recognition accuracy under endoscope NBI mode. BACKGROUND

[0002] In clinical practice, doctors will switch to NBI mode for observation as needed. The images obtained under this mode will exhibit darkening and grayish characteristics. However, when training the model, there are relatively few images in the database under NBI mode. Firstly, this is a relatively new technology, and secondly, in most cases in clinical practice, the ordinary mode is still used, which results in the inability to obtain a sufficient number of images under NBI mode for training. Generally, to enhance the generalization performance of the model, the prior art will use methods such as increasing random color perturbation when training the model, which can have certain improvements, but for NBI images with fixed characteristics, the effect is not very good. The problem caused by the unsatisfactory effect is that when the model is in inference, if the doctor turns on the NBI mode, the accuracy will be very low. SUMMARY

[0003] To solve the problems in the background art, the present application provides a method for improving target recognition accuracy under endoscope NBI mode, which comprises:

[0004] Under the HSV color space, the mean and mean square deviation of each channel of the image under NBI mode are calculated;

[0005] The image under non-NBI mode is converted to the HSV color space, and the pixel values of the image under non-NBI mode are normalized to the distribution of the mean and mean square deviation to obtain a first image under non-NBI mode;

[0006] The image under NBI mode and the first image under non-NBI mode are put into a preset model for training to obtain a target model.

[0007] Further, the calculation of the mean and mean square deviation of each channel of the image under NBI mode under the HSV color space is as follows:

[0008] Let the HSV value of an image under NBI mode at pixel point i be (h i , s i , v i ), and the mean (mean hx , mean sx , mean vx ) and mean square deviation of each channel of the image under NBI mode are calculated. As shown in equations (1) and (2);

[0009]

[0010] According to the mean and mean square deviation of each channel of the image under the NBI mode, the mean (mean h , mean s , mean v ) and mean square deviation As shown in equations (3) and (4);

[0011]

[0012] Further, the image under the non-NBI mode is converted in the HSV color space, and the pixel value of the image under the non-NBI mode is normalized to the distribution of the mean and mean square deviation, to obtain a first image under the non-NBI mode, specifically:

[0013] The image under the non-NBI mode is converted in the HSV color space;

[0014] The difference between the mean and the mean square value of the image under the non-NBI mode in the HSV color space and the mean of the image under the NBI mode in the HSV color space is calculated, as shown in equation (5);

[0015]

[0016] wherein, the mean of each channel of a given image under the non-NBI mode in the HSV color space is (mean sh , mean ss , mean sv ), and the mean square deviation is The mean of each channel of n images under the NBI mode in the HSV color space is (mean h , mean s , mean v ), and the mean square deviation is

[0017] The HSV value of each pixel point i* of the given image under the non-NBI mode is converted, as shown in equation (6);

[0018]

[0019] Further, the image under the non-NBI mode is converted in the HSV color space, and the pixel value of the image under the non-NBI mode is normalized to the distribution of the mean and mean square deviation, to obtain a first image under the non-NBI mode, specifically:

[0020] There are two ways:

[0021] Offline mode: normalize the pixel value of the image in non-NBI mode to the distribution of the mean and mean square deviation before training, to obtain the first image in non-NBI mode;

[0022] Online mode: randomly select an image in non-NBI mode during preset model training, normalize the pixel value of the image to the distribution of the mean and mean square deviation, to obtain the first image in non-NBI mode.

[0023] Further, the training of the NBI mode image and the first non-NBI mode image in the preset model to obtain the target model is specifically:

[0024] L1: input the NBI mode image and the first non-NBI mode image into the preset model;

[0025] L2: forward propagation is performed on the preset model to calculate the loss;

[0026] L3: according to the loss, the weight is updated by back propagation;

[0027] L4: repeat L2 and L3 to obtain the target model.

[0028] Further, before the training of the NBI mode image and the first non-NBI mode image in the preset model to obtain the target model, it further includes:

[0029] According to the image size standard of the preset model, the size of the NBI mode image and the first non-NBI mode image is adjusted;

[0030] The pixel value of the NBI mode image and the first non-NBI mode image is normalized.

[0031] A system for improving the target recognition accuracy of an endoscope in NBI mode, comprising:

[0032] A calculation unit for calculating the mean and mean square deviation of each channel of the NBI mode image in the HSV color space;

[0033] A pixel value conversion unit for converting the image in non-NBI mode in the HSV color space, and normalizing the pixel value of the image in non-NBI mode to the distribution of the mean and mean square deviation, to obtain the first image in non-NBI mode;

[0034] A model training unit for training the NBI mode image and the first non-NBI mode image in the preset model to obtain the target model.

[0035] Further, the pixel value conversion unit realizes two conversion modes:

[0036] Offline mode: normalize the pixel value of the image in non-NBI mode to the distribution of the mean and mean square deviation before training, to obtain the first image in non-NBI mode;

[0037] Online mode: randomly select the image in non-NBI mode during the preset model training, normalize the pixel value of the image to the distribution of the mean and mean square deviation, to obtain the first image in non-NBI mode.

[0038] Further, it also includes a preprocessing unit:

[0039] For the size adjustment of the image in NBI mode and the first image in non-NBI mode according to the image size standard of the preset model;

[0040] The pixel value of the image in NBI mode and the first image in non-NBI mode is normalized.

[0041] Beneficial effect: the present application provides a method for improving the target recognition accuracy in NBI mode of endoscope, calculates the mean and mean square deviation of each channel of the image in NBI mode in HSV color space, converts the image in non-NBI mode in HSV color space, normalizes the pixel value of the image in non-NBI mode to the distribution of the mean and mean square deviation, to obtain the first image in non-NBI mode, that is, to make the image in non-NBI mode have similar characteristics of the image in NBI mode, and then to have a large number of images in NBI mode, put the image in NBI mode and the first image in non-NBI mode into the preset model for training, to obtain the target model, thereby significantly improving the recognition accuracy of the target in NBI mode. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 It is a flowchart of a method for improving the target recognition accuracy in NBI mode of endoscope;

[0044] Figure 2 It is a target model training precision curve graph. DETAILED DESCRIPTION

[0045] In order to make those skilled in the art better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] In one embodiment, as shown in Figure 1 The present application provides a method for improving the target recognition accuracy in the NBI mode of an endoscope, which is applied to a system for improving the target recognition accuracy in the NBI mode of an endoscope. The system comprises a calculation unit, a pixel value conversion unit and a model training unit. The specific method comprises the following steps.

[0048] S1: In the HSV color space, the mean and the mean square deviation of each channel of the image in the NBI mode are calculated.

[0049] S2: The image in the non-NBI mode is converted in the HSV color space, and the pixel values of the image in the non-NBI mode are normalized to the distribution of the mean and the mean square deviation, to obtain a first image in the non-NBI mode.

[0050] S3: The image in the NBI mode and the first image in the non-NBI mode are put into a preset model for training, to obtain a target model.

[0051] Based on statistics, firstly, the mean and the variance of each channel of each image in the HSV color space are calculated for 1000 (the more, the better, limited by data and speed) images in the NBI mode, and finally the mean of the mean and the variance of each channel of all images is obtained.

[0052] Specifically, let the HSV value of an image in the NBI mode at a pixel point i be (h i , s i , v i ), the mean (mean hx , mean sx , mean vx ) and the mean square deviation of each channel of the image in the NBI mode are calculated.

[0053]

[0054] According to the mean and the mean square deviation of each channel of the image in the NBI mode, the mean (mean h , mean s , mean v ) and the mean square deviation of each channel of n images in the NBI mode are calculated. As shown in formula (3) and formula (4);

[0055]

[0056] Preferably, n non-NBI mode images are selected, and pixel value conversion processing is performed on each image. Specifically, the non-NBI mode image is converted in the HSV color space, and the difference between the mean value and the mean square value of each channel of the non-NBI mode image in the HSV color space and the NBI mode image in the HSV color space is calculated, as shown in formula (5);

[0057]

[0058] wherein, the mean value of each channel of a given non-NBI mode image in the HSV color space is (mean sh , mean ss , mean sv ) and the mean square error is The mean value of each channel of n NBI mode images in the HSV color space is (mean h , mean s , mean v ) and the mean square error is

[0059] The HSV value of each channel of a given non-NBI mode image is converted, as shown in formula (6);

[0060]

[0061] In the selection of the number of non-NBI mode images, the same or similar number of NBI mode images is selected, because in the case of a limited number of NBI mode images, in order to make the model training effect better and take into account the calculation cost.

[0062] Preferably, the NBI mode image and the first non-NBI mode image are put into the preset model for training to obtain a target model, specifically:

[0063] L1: the NBI mode image and the first non-NBI mode image are sent into the preset model;

[0064] L2: forward propagation is performed on the preset model to calculate the loss;

[0065] L3: according to the loss, the weights are updated by back propagation;

[0066] L4: repeat L2 and L3 to obtain the target model.

[0067] The preset model is composed of a network structure and weights, and the network structure is selected before starting training. For example, for a classification task, a VGG or ResNet model is selected, and for a target detection task, a YOLO model can be selected.

[0068] During the training process, L2 and L3 are repeated, as shown in the following table: Figure 2 When the number of repetitions reaches a certain number, the accuracy of the target model does not increase with the number of repetitions, but stabilizes at a certain value. In the judgment of the target model, a better accuracy can be considered as a better effect that the model can achieve under the current data, and can also be considered as the final effect (92%). Figure 2 In actual engineering, the loss is considered, and the model with the best accuracy is not necessarily selected, which is a common operation in engineering.

[0069] Weights are an important part of the target model, and weights are derived from input data, so the selection and processing of data are related to the effect of the target model. In this embodiment, there are two ways to achieve data selection and processing,

[0070] The first way is offline mode: data selection and processing before training. From the data set, especially from some classes with few NBI patterns, select some non-NBI pattern images, convert them to HSV color space, and use the mean and variance of each channel of the NBI pattern image in the HSV color space as described above. Normalize the pixel values of the non-NBI pattern image to the mean and variance distribution to make it have similar characteristics to the NBI pattern image.

[0071] The second way is online mode: put the non-NBI pattern images and NBI pattern images without processing into the model for training. During the training process, a probability (the probability is selected between 0 and 1) can be set, and the corresponding number of non-NBI pattern images are randomly selected according to the probability. Process the selected images using the processing method for non-NBI pattern images in offline mode to make the selected images have similar characteristics to the NBI pattern image.

[0072] Preferably, before training the model, the images input to the model need to be processed in size and pixel value normalization, for example, for input images, first adjust the size of the input image to meet the requirements of the model, such as 224x224 (h*w), and normalize the pixel value by dividing by 255 to [0, 1].

[0073] A system for improving the accuracy of target recognition in endoscopic NBI mode, comprising:

[0074] A calculating unit is configured to calculate the mean value and the mean square deviation of each channel of the image in the NBI mode in the HSV color space;

[0075] A pixel value converting unit is configured to convert the image in the non-NBI mode to the HSV color space, normalize the pixel value of the image in the non-NBI mode to the distribution of the mean value and the mean square deviation, and obtain a first image in the non-NBI mode;

[0076] A model training unit is configured to train the image in the NBI mode and the first image in the non-NBI mode in a preset model, and obtain a target model.

[0077] Preferably, the pixel value converting unit realizes two conversion modes:

[0078] An offline mode: the pixel value of the image in the non-NBI mode is normalized to the distribution of the mean value and the mean square deviation before training, and the first image in the non-NBI mode is obtained;

[0079] An online mode: an image in any mode is randomly selected during the training of the preset model, the pixel value of the image is normalized to the distribution of the mean value and the mean square deviation, and the first image in the non-NBI mode is obtained.

[0080] Preferably, the system further comprises a preprocessing unit:

[0081] The preprocessing unit is configured to perform size adjustment on the image in the NBI mode and the first image in the non-NBI mode according to the image size standard of the preset model;

[0082] The preprocessing unit is configured to perform normalization processing on the pixel value of the image in the NBI mode and the first image in the non-NBI mode.

[0083] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), ultra-scale X DRAM (UXDRAM), and others.

[0084] SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRAM), and the like.

[0085] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered to be within the scope of the present specification.

[0086] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A method for improving target recognition accuracy in endoscopic NBI mode, characterized in that, The method includes: In the HSV color space, calculate the mean and standard deviation of each channel of the image in NBI mode; The image in non-NBI mode is converted to HSV color space, and the pixel values ​​of the image in non-NBI mode are normalized to the distribution of the mean and the mean square error to obtain the first image in non-NBI mode. Specifically, the image in non-NBI mode is converted to HSV color space, and the difference between the mean and the mean square error of each channel of the image in non-NBI mode in HSV color space and the image in NBI mode in HSV color space is calculated, as shown in equation (5). Here, let the mean value of each channel of a given image in non-NBI mode in the HSV color space be (mean). sh mean ss mean sv ) and mean square error Let the mean value of each channel of n images in NBI mode in the HSV color space be (mean...). h mean s mean v ) and mean square error The HSV channel values ​​of each pixel i* of the given non-NBI mode image are converted as shown in equation (6); The images in NBI mode and the first non-NBI mode are fed into a preset model for training to obtain the target model.

2. The method for improving target recognition accuracy in endoscopic NBI mode according to claim 1, characterized in that, The process involves converting the image in non-NBI mode to the HSV color space and normalizing the pixel values ​​of the image in non-NBI mode to the distribution of the mean and standard deviation, resulting in the first image in non-NBI mode. There are two methods: Offline mode: Before training, the pixel values ​​of the image in non-NBI mode are normalized to the distribution of the mean and standard deviation to obtain the first image in non-NBI mode. Online mode: During the training of the preset model, images in non-NBI mode are randomly selected, and the pixel values ​​of the images are normalized to the distribution of the mean and standard deviation to obtain the first image in non-NBI mode.

3. The method for improving target recognition accuracy in endoscopic NBI mode according to claim 1, characterized in that, The specific steps for training the target model by feeding the image in NBI mode and the image in the first non-NBI mode into a preset model are as follows: L1: Input the image in NBI mode and the first image in non-NBI mode into the preset model; L2: Perform forward propagation on the preset model and calculate the loss; L3: Update the weights by backpropagation based on the loss; L4: Repeat L2 and L3 to obtain the target model.

4. The method for improving target recognition accuracy in endoscopic NBI mode according to claim 1, characterized in that, Before training the target model by feeding the image in NBI mode and the image in the first non-NBI mode into the preset model, the process further includes: Based on the image size standard of the preset model, the image in the NBI mode and the image in the first non-NBI mode are resized. The pixel values ​​of the image in the NBI mode and the image in the first non-NBI mode are normalized.

5. A system for improving target recognition accuracy in endoscopic NBI mode, characterized in that, include: Calculation unit: Used to calculate the mean and mean square error of each channel of an image in NBI mode under HSV color space; Pixel value conversion unit: used to convert the image in non-NBI mode to HSV color space, normalize the pixel values ​​of the image in non-NBI mode to the distribution of the mean and mean square error, and obtain the first image in non-NBI mode. Specifically, the image in non-NBI mode is converted to HSV color space, and the difference between the mean and mean square error of each channel of the image in non-NBI mode in HSV color space and the image in NBI mode in HSV color space is calculated, as shown in equation (5). Here, let the mean value of each channel of a given image in non-NBI mode in the HSV color space be (mean). sh mean ss mean sv ) and mean square error Let the mean value of each channel of n images in NBI mode in the HSV color space be (mean...). h mean s mean v ) and mean square error The HSV channel values ​​of each pixel i* of the given non-NBI mode image are converted as shown in equation (6); Model training unit: The images in NBI mode and the first non-NBI mode are fed into the preset model for training to obtain the target model.

6. The system for improving target recognition accuracy in endoscopic NBI mode according to claim 5, characterized in that, The pixel value conversion unit implements two conversion methods: Offline mode: Before training, the pixel values ​​of the image in non-NBI mode are normalized to the distribution of the mean and standard deviation to obtain the first image in non-NBI mode. Online mode: During the training of the preset model, images in non-NBI mode are randomly selected, and the pixel values ​​of the images are normalized to the distribution of the mean and standard deviation to obtain the first image in non-NBI mode.

7. The system for improving target recognition accuracy in endoscopic NBI mode according to claim 5, characterized in that, It also includes a preprocessing unit: This is used to adjust the size of the image in the NBI mode and the image in the first non-NBI mode according to the image size standard of the preset model; The pixel values ​​of the image in the NBI mode and the image in the first non-NBI mode are normalized.

Images