Endoscopic image processing method and system

Abstract

The present invention belongs to the field of image processing technology, and discloses an endoscopic image processing method, comprising: acquiring an endoscopic image and endoscopic data corresponding to an endoscope; performing image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image; performing feature extraction on the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature subgraphs; performing image processing on the obtained feature subgraphs to obtain corresponding feature enhanced images, and performing image fusion processing based on the feature enhanced images to obtain corresponding high-quality images; performing image quality evaluation on the obtained high-quality image, and judging whether the high-quality image meets the requirements based on the image quality evaluation result; the present invention shows significant beneficial effects in improving image quality, optimizing image processing flow, etc.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to an endoscope image processing method and system. Background Art

[0002] The patent with application publication number CN117204790B discloses an image processing method and system for endoscope, which relates to the field of image processing technology. The system includes a device layer, a control layer, and a data storage layer; the device layer includes: an endoscope camera, an endoscope light source, and a CPU; the control layer includes: an endoscope preprocessing module, an image acquisition module, an image processing module, and a feedback module; the endoscope preprocessing module is used to perform anti-fogging preprocessing on the endoscope; the image acquisition module is used to collect images of the inside of the intestine and transmit the images to the CPU for image processing; the image processing module is used to perform in-depth analysis on the images received by the CPU; and the feedback module is used to feed back the in-depth analysis results of the images to the medical staff performing endoscopic surgery. The present invention can facilitate medical staff to quickly and accurately find the lesion during the subsequent review process.

[0003] Compared with the existing technology, the traditional endoscopic image processing method may cause image blur, low contrast, color distortion or noise due to factors such as lighting conditions, camera angle, reflective characteristics of internal tissues, movement in the patient's body, etc., and there are also many problems such as insufficient noise processing and difficulty in removing highlights.

[0004] In view of this, the present invention proposes an endoscopic image processing method and system to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0006] An endoscope image processing method, comprising:

[0007] Step 1: obtaining an endoscopic image group and endoscopic data corresponding to the endoscope; the endoscopic image group is composed of a plurality of endoscopic images;

[0008] Step 2: performing image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image;

[0009] Step 3: extracting features from the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs;

[0010] Step 4: performing image processing on the obtained feature sub-image to obtain a corresponding feature enhanced image, and performing image fusion processing based on the feature enhanced image to obtain a corresponding high-quality image;

[0011] Step 5: Perform image quality assessment on the obtained high-quality image to obtain an image assessment result, and determine whether the high-quality image meets the requirements based on the image assessment result. If it does, the corresponding high-quality image is used as output; if it does not, return to step 3.

[0012] Furthermore, the process of obtaining the endoscopic image group and endoscopic data corresponding to the endoscope includes:

[0013] Setting a data acquisition unit, wherein the data acquisition unit is composed of a data acquisition terminal and an image acquisition terminal;

[0014] When the data acquisition unit recognizes that the corresponding endoscope recognizes the generation of the video stream, the data acquisition unit collects the endoscopic data and the video stream corresponding to the corresponding endoscope based on the data acquisition terminal and the image acquisition terminal respectively;

[0015] At the same time, the image acquisition terminal reads the acquired video stream and divides it into a number of continuous single-frame images based on the acquisition time; and compares the acquired single-frame images with the preset screening criteria; if the corresponding single-frame image meets the screening criteria, the corresponding single-frame image is marked as an endoscopic image; if it does not meet the screening criteria, it is discarded;

[0016] The marked endoscopic images are counted and sorted based on corresponding acquisition times to obtain corresponding endoscopic image groups.

[0017] Furthermore, the obtained endoscopic image is subjected to image preprocessing, and the process of obtaining the corresponding preprocessed image includes:

[0018] Performing an initial image transformation on the obtained endoscopic image, and marking the endoscopic image after the initial image transformation as an initial image;

[0019] Perform image decomposition on the marked initial image, and obtain first channel images c corresponding to the corresponding RGB color channels respectively;

[0020] Performing initial image transformation on the obtained first channel image c to obtain a corresponding blurred image MH;

[0021] Constructing a corresponding pixel feature matrix based on the first channel image c; performing a convolution operation on the pixel feature matrix and a pre-selected Gaussian function to obtain a corresponding Gaussian matrix, and obtaining eigenvalues ​​λ1 and λ2 corresponding to the Gaussian matrix;

[0022] Based on the eigenvalues ​​λ1 and λ2, pixel values ​​of the pixels in the corresponding blurred image MH are adjusted;

[0023] The pixel-adjusted blurred image MH is inversely transformed to obtain a second channel image corresponding to the corresponding RGB channel; then, the obtained second channel image is image fused to obtain a corresponding preprocessed image.

[0024] Furthermore, the formula for the initial image transformation is: Where V out Indicates the output image after the initial image transformation is completed, V in represents the input image participating in the initial image transformation, γ is the brightness correction factor;

[0025] The corresponding formula for initial image transformation is: In the formula, I c (i, j) represents the pixel value at the pixel point (i, j) in the first channel image c; V MH (i, j) represents the pixel value at the pixel point (i, j) in the blurred image MH;

[0026] ε and I cmin Respectively represent the variance and minimum pixel value of the pixel values ​​in the first channel image c;

[0027] The inversion formula is: Where V2_ c (i, j) represents the pixel value at the pixel point (i, j) in the second channel image; I min (i,j) represents the minimum pixel value at pixel point (i,j).

[0028] Furthermore, the process of extracting features from the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs includes:

[0029] Reading the obtained preprocessed image, and importing the obtained preprocessed image as an input image into a pre-built improved feature model to obtain a corresponding model output result, wherein the model output result includes a plurality of output features corresponding to the preprocessed image;

[0030] Obtaining the pixel position of the corresponding output feature in the preprocessed image, and marking the pixel point corresponding to the pixel position as a feature point;

[0031] A blank window is constructed, the feature point is used as the center point of the blank window, and the preprocessed image is intercepted based on the feature point to obtain a corresponding feature sub-image.

[0032] Furthermore, the construction process of the improved feature model is as follows:

[0033] The network framework of the improved feature model is defined as a ResNet convolutional neural network with an attention mechanism; the improved feature model includes an attention layer and a perception layer; the attention layer includes a channel attention sublayer and a spatial attention sublayer, and both the attention layer and the perception layer are composed of a convolution layer and a pooling layer, and the perception layer also includes a ReLU activation function;

[0034] Input the obtained input image into the corresponding channel attention sublayer, and the channel attention sublayer performs maximum pooling operation and average pooling operation on the corresponding input image respectively; obtain the corresponding pooling feature map; input the obtained pooling feature map into the perception layer, and the perception layer performs convolution processing on the corresponding pooling feature map respectively to obtain the corresponding convolution feature map;

[0035] The formula for convolution processing is: R0 = η (W SFE ×I LR +B); where η() represents the ReLU activation function in the corresponding perception layer; W SFE Represents the weight value, I LR represents the input pooling feature map; B represents the bias term; R0 represents the output convolution feature map;

[0036] The obtained convolution feature map is added bit by bit; and the corresponding weight coefficient Mc is obtained based on the Sigmoid activation function;

[0037] Based on the weight coefficient Mc, the corresponding input image is bitwise multiplied to obtain the corresponding channel attention feature;

[0038] Based on the spatial attention sublayer, the corresponding channel attention features are subjected to maximum pooling and average pooling operations to obtain two spatial feature maps, and the two feature maps are concatenated according to the channels to obtain corresponding channel feature maps;

[0039] After performing a convolution operation on the obtained channel feature map, a weight factor Ms is obtained after passing through a Sigmoid activation function; based on the weight factor Ms, the corresponding channel attention feature is bitwise multiplied to obtain the corresponding output feature;

[0040] Acquire a historical endoscopic data set, wherein the endoscopic data set includes a plurality of historical endoscopic images and feature labels corresponding to the corresponding endoscopic images;

[0041] Performing augmentation processing on the obtained historical endoscopic images, and constructing corresponding training data based on the augmented endoscopic data set;

[0042] The obtained training data is divided into several training batches, and corresponding pre-training weights are assigned to the corresponding training batches; the training batches with pre-training weights are input into the constructed improved feature model for iterative training until the model parameters of the corresponding improved feature model converge after P1 consecutive training batches, and the training is stopped, and the corresponding improved feature model is output; P1 is a natural number.

[0043] Furthermore, the process of performing image processing on the obtained feature sub-image to obtain a corresponding feature enhanced image, and performing image fusion processing based on the feature enhanced image to obtain a corresponding high-quality image includes:

[0044] Based on the wavelet threshold algorithm, the obtained feature sub-image is decomposed to obtain the corresponding high-frequency feature image and low-frequency feature image;

[0045] Performing grayscale conversion on the low-frequency feature image to obtain a corresponding low-frequency grayscale image, and then obtaining the grayscale value corresponding to each pixel in the corresponding low-frequency grayscale image, and constructing a corresponding grayscale histogram based on the grayscale value;

[0046] Performing adaptive grayscale equalization processing on the obtained low-frequency grayscale image to obtain a corresponding enhanced low-frequency image;

[0047] Performing image enhancement on the obtained high-frequency feature image based on an improved wavelet threshold function to obtain an enhanced high-frequency image;

[0048] Perform wavelet reconstruction on the enhanced low-frequency image and the enhanced high-frequency image to obtain the corresponding enhanced feature image;

[0049] Counting all enhanced feature images, and performing image stitching on the obtained enhanced feature images based on the inverse process of image interception to obtain the corresponding endoscopic enhanced image;

[0050] Reading endoscopic data corresponding to a corresponding endoscopic image at a corresponding acquisition time; and acquiring a corresponding image mapping correction coefficient based on the endoscopic data;

[0051] Inputting the obtained endoscopy enhanced image into the constructed improved feature model table to obtain feature points corresponding to the corresponding endoscopy enhanced image, and marking the corresponding feature points as second feature points;

[0052] At the same time, the feature point corresponding to the corresponding endoscopic image is read and marked as the first feature point;

[0053] The Euclidean distances between different first feature points and second feature points are obtained respectively, and at the same time, the image similarity of the feature sub-graphs corresponding to the corresponding first feature points and the second feature points is obtained; and based on the similarity, it is determined whether the corresponding first feature points and the second feature points are matching feature points; if they are matching feature points, the corresponding point association relationship is established; if they are not matching feature points, no other operations are performed;

[0054] Further, based on the obtained point correlation relationship and the obtained image mapping correction coefficient, point correction is performed on the second characteristic point in the corresponding endoscopic enhanced image;

[0055] The endoscopic enhanced image after point correction is weightedly summed with the corresponding endoscopic image to obtain a corresponding high-quality image.

[0056] Furthermore, the corresponding formula for adaptive grayscale balancing is: Wherein, H_f(t) represents the t-th grayscale value after adaptive grayscale equalization in the corresponding grayscale histogram; f(t) represents the t-th grayscale value in the corresponding grayscale histogram; L represents the part of the histogram that exceeds F; F represents the limit threshold, and the corresponding limit threshold F=Avg-[a×(Avg×Num(t)-Avg)]; where Avg represents the average grayscale value; a represents the adjustable coefficient, a∈(0,1); Num(t) represents the grayscale level corresponding to the t-th grayscale value in the grayscale histogram;

[0057] The formula of the improved wavelet threshold function is:

[0058]

[0059] Where V_g(V_p(x,y)) represents the enhanced high-frequency image after being processed by the improved wavelet threshold function; V_p(x,y) represents the high-frequency feature image; |V_p(x,y)| represents the absolute value of the wavelet coefficient corresponding to the pixel point (x,y) in the high-frequency feature image;

[0060] T represents the adaptive wavelet threshold, where Where M and N represent the length and width of the high-frequency feature image, respectively; s and m represent the preset scale factor and adjustment factor.

[0061] Furthermore, the process of evaluating the image quality of the obtained high-quality image and judging whether it meets the requirements based on the evaluation result includes:

[0062] The high-quality image is compared with the original endoscopic image to obtain a corresponding comparison result, and based on the comparison result, it is determined whether the corresponding high-quality image meets the requirements. If it does not meet the requirements, it is marked as an image to be inspected; if it does not meet the requirements, the corresponding high-quality image is fed back to step three as a preprocessing image until it meets the requirements;

[0063] The obtained images to be checked are counted, and the images to be checked are sorted based on the corresponding acquisition time to obtain a corresponding check image group, and the group is fed back to the responsible doctor of the corresponding patient.

[0064] Furthermore, an endoscope image processing system comprises:

[0065] A data acquisition module, used for acquiring an endoscopic image group and endoscopic data corresponding to the endoscope; the endoscopic image group is composed of a plurality of endoscopic images;

[0066] An image preprocessing module, used to perform image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image;

[0067] Image processing module; used for performing feature extraction on the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs; performing image processing on the obtained feature sub-graphs to obtain corresponding feature enhanced images, and performing image fusion processing based on the feature enhanced images to obtain corresponding high-quality images;

[0068] Image evaluation module: performs image quality evaluation on the obtained high-quality image, obtains the corresponding image evaluation result, and judges whether the high-quality image meets the requirements based on the image evaluation result. If it meets the requirements, the corresponding high-quality image is used as output; if it does not meet the requirements, it is returned to the image processing module.

[0069] The technical effects and advantages of the endoscopic image processing method and system of the present invention are as follows: through the steps of image preprocessing, feature extraction, feature subgraph processing, feature enhancement and image fusion, the quality of the endoscopic image is significantly improved; in particular, through adaptive grayscale equalization processing and improved wavelet threshold function, the contrast and detail features of the image are enhanced, the noise is reduced, and the image is clearer and more accurate; at the same time, the image processing process is simplified, manual intervention is reduced, and processing efficiency and stability are improved. At the same time, by presetting the screening criteria and image quality evaluation, it is ensured that the quality of the output image meets the requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of a process of an endoscope image processing method of the present invention;

[0071] Figure 2 The figure is a schematic diagram of the principle of an endoscope image processing system of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Example 1

[0074] See also Figure 1 As shown, the endoscope image processing method described in this embodiment includes:

[0075] Step 1: obtaining an endoscopic image group and endoscopic data corresponding to the endoscope; the endoscopic image group is composed of a plurality of endoscopic images;

[0076] Step 2: performing image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image;

[0077] Step 3: extracting features from the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs;

[0078] Step 4: performing image processing on the obtained feature sub-image to obtain a corresponding feature enhanced image, and performing image fusion processing based on the feature enhanced image to obtain a corresponding high-quality image;

[0079] Step 5: Perform image quality assessment on the obtained high-quality image to obtain an image assessment result, and determine whether the high-quality image meets the requirements based on the image assessment result. If it meets the requirements, mark the corresponding high-quality image as an image to be inspected; if it does not meet the requirements, return to step 3;

[0080] It should be further explained that, in the specific implementation process, the process of obtaining the endoscopic image group and endoscopic data corresponding to the endoscope includes:

[0081] A data acquisition unit is provided, wherein the data acquisition unit is composed of a data acquisition terminal and an image acquisition terminal; and the data acquisition unit is connected to an output end of a corresponding endoscope via a network port;

[0082] Slowly inserting the endoscope into the area to be detected of the patient, and performing video acquisition of the corresponding area to be detected based on the endoscope, and generating a corresponding video stream;

[0083] Furthermore, when the data acquisition unit recognizes the generation of the corresponding video stream, the data acquisition terminal collects the operation information of the corresponding endoscope during the video acquisition process based on the corresponding network port to obtain the corresponding endoscope data, wherein the endoscope data includes the endoscope position, moving direction, moving speed, jitter offset value and specification parameters;

[0084] At the same time, the image acquisition terminal reads the video stream generated by the corresponding endoscope, and divides it into a number of continuous single-frame images based on the acquisition time; and compares the obtained single-frame images with the preset screening criteria; if the corresponding single-frame image meets the screening criteria, the corresponding single-frame image is marked as an endoscopic image; if it does not meet the screening criteria, it is discarded; wherein the preset screening criteria include resolution, contrast, sharpness, color performance, noise and distortion, and visual perception, etc.;

[0085] Counting the marked endoscopic images and sorting them based on corresponding acquisition times to obtain corresponding endoscopic image groups;

[0086] It should be further explained that, in the specific implementation process, when the image acquisition terminal recognizes the generation of an endoscopic image group, it will calculate the time deviation of the acquisition time corresponding to the adjacent endoscopic images in the corresponding endoscopic image group to obtain the corresponding loss time; and compare it with the preset time threshold. If the loss time is less than the time threshold, no other operation is performed. If the loss time is greater than or equal to the time threshold, indicating that there is an image acquisition gap in the obtained endoscopic image group, a collection failure notification is fed back to the patient's responsible doctor, and the responsible doctor adjusts the corresponding endoscope and re-collects the corresponding endoscopic image group.

[0087] It should be further explained that, in the specific implementation process, the obtained endoscopic image is preprocessed to obtain the corresponding preprocessed image, which includes:

[0088] Performing an initial image transformation on the obtained endoscopic image, wherein the initial image transformation is used to preliminarily adjust the image brightness of the corresponding endoscopic image, for example, if the image brightness is high light, then appropriately reducing the brightness of the corresponding endoscopic image; if the image brightness is low light, then appropriately enhancing the image brightness of the corresponding endoscopic image;

[0089] Among them, the formula for the initial image transformation is: Where V out Indicates the output image after the initial image transformation is completed, V in represents the input image involved in the initial image transformation, and γ is the brightness correction factor. By adjusting the brightness correction factor, the brightness value of the image is changed so that it can be presented in the most appropriate range, thereby achieving the purpose of changing the contrast and brightness of the endoscopic image. Among them, In the formula, L_(V in ) represents the mean brightness of the endoscope image; Li(V in ) represents the brightness value corresponding to the i-th pixel in the endoscopic image, i=1, 2, ..., N, N>0 and N is an integer, and N represents the total number of pixels in the corresponding endoscopic image;

[0090] Marking the endoscopic image after the transformation of the initial image as the initial image;

[0091] Further, the initial image is decomposed to obtain the first channel images c corresponding to the corresponding RGB color channels respectively; wherein c∈(R, G, B);

[0092] The obtained first channel image c is subjected to initial image transformation to obtain a corresponding blurred image MH, where MH=c; the corresponding initial image transformation formula is: In the formula, I c (i, j) represents the pixel value at the pixel point (i, j) in the first channel image c; V MH (i, j) represents the pixel value at the pixel point (i, j) in the blurred image MH;

[0093] ε and I cmin Respectively represent the variance of pixel values ​​and the minimum pixel value in the corresponding first channel image c; wherein the initial image transformation is used to highlight certain important features in the image, such as edges, textures, etc., while suppressing unimportant information, so that the image is more suitable for further analysis and processing; and helps to simplify the problem and reduce the amount of calculation;

[0094] Based on the first channel image c, a corresponding pixel feature matrix is ​​constructed; and the pixel feature matrix is ​​subjected to a Gaussian convolution operation with a pre-selected Gaussian function to obtain a corresponding Gaussian matrix, and eigenvalues ​​λ1 and λ2 corresponding to the Gaussian matrix are obtained; wherein the process of performing a Gaussian convolution operation with the pre-selected Gaussian function is an existing process, and the present invention will not elaborate on it in detail;

[0095] Based on the eigenvalues ​​λ1 and λ2, pixel adjustment is performed on the pixel value at the pixel point (i, j) in the corresponding blurred image MH. The corresponding pixel adjustment process is as follows: reading the obtained eigenvalue λ1;

[0096] If λ1≤0, the pixel value at the pixel point (i, j) in the corresponding blurred image MH is retained, V′ MH (i,j)=V MH (i, j); if λ1>0, the pixel value at the pixel point (i, j) in the corresponding blurred image MH is adjusted based on the eigenvalues ​​λ1 and λ2, and the corresponding adjustment formula is: Where V′ MH (i, j) represents the adjusted pixel value at the pixel point (i, j); β represents a membership parameter used to control the improvement ability of the linear detail performance in the endoscopic image. In the present invention, the default value is β = 0.5;

[0097] It should be further explained that, in the specific implementation process, the process of constructing the pixel feature matrix includes:

[0098] Read the obtained first channel image c, obtain the corresponding pixel point (i, j) in the first channel image c; and calculate the partial derivative of the corresponding pixel point (i, j), and obtain the corresponding pixel feature matrix based on it In the formula, H_I c (i, j) represents the pixel feature matrix corresponding to the pixel point (i, j) in the first channel image c; I ii ,I jj Respectively represent the second-order partial derivatives of the corresponding pixel point (i, j) in the horizontal and vertical directions; I ij ,I ji Represents the second-order partial derivative of the corresponding pixel point (i, j) mixed in the horizontal and vertical directions;

[0099] Based on the adjustment process of the corresponding pixel values, the pixel values ​​corresponding to each pixel point in the corresponding blurred image MH are adjusted until the adjustment is completed; the corresponding blurred image MH is reversely transformed to obtain the second channel image corresponding to the corresponding RGB channel; then, the obtained second channel image is channel-fused to obtain the corresponding pre-processed image, wherein the channel fusion refers to re-fusion of the second channel image corresponding to the RGB three-color channel into an image containing the RGB three-color channels at the same time;

[0100] The inversion formula is: Where V2_ c (i, j) represents the pixel value at the pixel point (i, j) in the second channel image; I min (i, j) represents the minimum pixel value at pixel point (i, j); V′ MH (i, j) represents the pixel value at the pixel point (i, j) in MH after pixel adjustment and blurring of the image.

[0101] It should be further explained that, in a specific implementation process, the process of extracting features from the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs includes:

[0102] Reading the obtained preprocessed image, and importing the obtained preprocessed image as an input image into a pre-built improved feature model to obtain a corresponding model output result, wherein the model output result includes a number of output features corresponding to the corresponding preprocessed image;

[0103] Obtaining the pixel position of the corresponding output feature in the preprocessed image, and marking the pixel point corresponding to the pixel position as a feature point;

[0104] Constructing a blank window, taking the feature point as the center point of the blank window, and performing image interception on the preprocessed image based on the feature point to obtain a plurality of feature subgraphs; wherein the number of the feature subgraphs corresponds to the encounter feature points, and each feature subgraph includes at most one other feature point except the feature point corresponding to the center point;

[0105] It should be further explained that, in the specific implementation process, the image interception refers to locally intercepting the pre-processed image with the feature point as the center and the blank window as the size;

[0106] It should be further explained that, in the specific implementation process, the construction process of the improved feature model is:

[0107] The network framework of the improved feature model is defined as a ResNet convolutional neural network with an attention mechanism; it is used to identify feature information of different levels of the input image and perform feature fusion to obtain output features corresponding to the input image; wherein the attention mechanism is an efficient mechanism for optimizing feature extraction of convolutional neural networks by combining channel and spatial attention;

[0108] The process of obtaining the corresponding output features in the improved feature network includes:

[0109] The improved feature model includes an attention layer and a perception layer; wherein the attention layer includes a channel attention sublayer and a spatial attention sublayer, and both the attention layer and the perception layer include a convolution layer and a pooling layer, and the perception layer also includes a ReLU activation function;

[0110] The obtained input image is input into the corresponding channel attention sublayer, and the channel attention sublayer performs the maximum pooling operation and the average pooling operation on the corresponding input image respectively; the corresponding pooling feature map is obtained; the obtained pooling feature map is input into the perception layer, and the perception layer performs convolution processing on the corresponding pooling feature map respectively to obtain the corresponding convolution feature map, wherein the convolution processing formula is: R0 = η (W SFE ×I LR +B); where η() represents the ReLU activation function in the corresponding perception layer; W SFE Represents the weight value, ILR represents the input pooling feature map; B represents the bias term; R0 represents the output convolution feature map;

[0111] Then, the obtained convolution feature map is bitwise added; and the corresponding weight coefficient Mc is obtained based on the Sigmoid activation function; then, the corresponding input image is bitwise multiplied based on the weight coefficient Mc to obtain the corresponding channel attention feature;

[0112] Based on the spatial attention sublayer, the corresponding channel attention features are subjected to maximum pooling and average pooling operations respectively to obtain two spatial feature maps, and the two feature maps are spliced ​​by channel to obtain the corresponding channel feature map, and then the obtained channel feature map is subjected to convolution operation, and then the weight factor Ms is obtained after the Sigmoid activation function; based on the weight factor Ms, the corresponding channel attention features are bitwise multiplied to obtain the corresponding output features; wherein the Sigmoid activation function and the ReLU activation function are both selected in advance according to actual needs;

[0113] Acquire a historical endoscopic data set, wherein the endoscopic data set includes a plurality of historical endoscopic images and feature labels corresponding to the corresponding endoscopic images, wherein the feature labels are feature points manually marked in some historical endoscopic images by experienced gastrointestinal endoscopists;

[0114] Perform augmentation processing on the obtained historical endoscopic images, and after the processing is completed, construct corresponding training data based on the endoscopic data; wherein the augmentation processing refers to performing image processing on the corresponding historical endoscopic images using conventional data augmentation operations such as rotation, scaling, symmetry, etc., which can not only increase the model training amount for a single training, but also randomly generate small targets during the random cropping process, thereby enhancing the image feature extraction capability of the model;

[0115] The obtained training data is divided into multiple training batches, and corresponding pre-training weights are assigned to the corresponding training batches; the training batches with pre-training weights are input into the constructed improved feature model for iterative training until the model parameters of the corresponding improved feature model converge after P1 consecutive training batches, and the training is stopped, and the corresponding improved feature model is output; P1 is a natural number;

[0116] It should be further explained that, in a specific implementation process, image processing is performed on the obtained feature sub-image to obtain a corresponding feature enhanced image, and image fusion processing is performed based on the feature enhanced image to obtain a corresponding high-quality image. The process includes:

[0117] Based on the wavelet threshold algorithm, the obtained feature sub-image is decomposed to obtain the corresponding high-frequency feature image and low-frequency feature image;

[0118] The obtained low-frequency feature map is converted into grayscale to obtain a corresponding low-frequency grayscale image, and then the grayscale value corresponding to each pixel in the corresponding low-frequency grayscale image is obtained, and a corresponding grayscale histogram is constructed based on the grayscale value;

[0119] Performing adaptive grayscale equalization processing on the obtained low-frequency grayscale image to obtain a corresponding enhanced low-frequency image to further enhance the contrast of the image;

[0120] The corresponding formula for adaptive grayscale balancing is:

[0121] Where H_f(t) represents the tth grayscale value in the corresponding grayscale histogram after adaptive grayscale equalization processing;

[0122] f(t) represents the tth grayscale value in the corresponding grayscale histogram; where t is an integer and t∈(0,255);

[0123] L represents the part of the histogram that exceeds F;

[0124] F(t) represents the limit threshold corresponding to the gray value t, and the corresponding limit threshold is: F(t) = Avg-[a×(Avg×Num(t)-Avg)]; where Avg represents the average gray value; a represents the adjustable coefficient, a∈(0,1); Num(t) represents the gray level corresponding to the t-th gray value in the gray histogram;

[0125] Reading the obtained high-frequency feature image, and performing image enhancement on the obtained high-frequency feature image based on the improved wavelet threshold function to obtain an enhanced high-frequency image, so as to reduce noise in the high-frequency feature image and further enhance detail features of the high-frequency feature image;

[0126] The formula of the improved wavelet threshold function is:

[0127]

[0128] Where V_g(V_p(x,y)) represents the enhanced high-frequency image after being processed by the improved wavelet threshold function; V_p(x,y) represents the high-frequency feature image; |V_p(x,y)| represents the absolute value of the wavelet coefficient corresponding to the pixel point (x,y) in the high-frequency feature image;

[0129] T represents the adaptive wavelet threshold, where Where M and N represent the length and width of the high-frequency feature image respectively; max{} represents the maximum value;

[0130] s and m represent the preset scale factor and adjustment factor; among them, m is used to improve the stability of the wavelet threshold function, so that the curve becomes gentle when the function is close to the adaptive wavelet threshold, and a part of the wavelet coefficients is retained as much as possible, which effectively reduces the loss of high-frequency detail information of the image and prevents excessive noise reduction;

[0131] The obtained enhanced low-frequency image and enhanced high-frequency image are read and reconstructed by wavelet to obtain a corresponding enhanced feature image;

[0132] All enhanced feature images are counted, and based on the image interception process of the corresponding pre-processed image in the corresponding blank window, the obtained enhanced feature images are stitched to obtain the corresponding endoscopic enhanced image; wherein, for the overlapping part of adjacent enhanced feature images, the average value of the pixel values ​​corresponding to the pixel points of the overlapping part is obtained as the final pixel value at the corresponding pixel point after stitching;

[0133] Further, the endoscopic data corresponding to the corresponding endoscopic image at the corresponding acquisition time is read; and the corresponding image mapping correction coefficient is obtained based on the endoscopic data, and the image mapping correction coefficient can be used to represent the mapping relationship between the target endoscope and the characteristic points in the corresponding endoscopic image at the corresponding acquisition time; the acquisition process is the prior art, and the present invention will not elaborate on it in detail;

[0134] Inputting the obtained endoscopy enhanced image into the constructed improved feature model table to obtain feature points corresponding to the corresponding endoscopy enhanced image and marking them as second feature points;

[0135] At the same time, the feature point corresponding to the corresponding endoscopic image is read and marked as the first feature point;

[0136] The Euclidean distances between different first feature points and second feature points are obtained respectively, and at the same time, the image similarity of the feature subgraphs corresponding to the corresponding first feature points and the second feature points is obtained; and based on the similarity, whether the corresponding first feature points and the second feature points are matching feature points is determined; if they are matching feature points, the corresponding point association relationship is constructed; if they are not matching feature points, no other operations are performed; wherein, the image similarity refers to the evaluation of the similarity of the corresponding feature subgraph brackets from multiple angles such as the structural similarity index (SSIM), peak signal-to-noise ratio (PSNR) and perceptual hashing algorithm, such as brightness, contrast, structural information, etc.;

[0137] Further, based on the obtained point correlation relationship and the obtained image mapping correction coefficient, point correction is performed on the second characteristic point in the corresponding endoscopic enhanced image;

[0138] The endoscopic enhanced image after point correction is weightedly summed with the corresponding endoscopic image to obtain a corresponding high-quality image.

[0139] It should be further explained that, in the specific implementation process, the process of evaluating the image quality of the obtained high-quality image and judging whether it meets the requirements based on the evaluation results includes:

[0140] The obtained high-quality image is read and compared with the original endoscopic image, and based on the comparison result, it is determined whether the corresponding high-quality image meets the requirements. If it does not meet the requirements, it is marked as an image to be inspected; if it does not meet the requirements, the corresponding high-quality image is fed back to step three as a pre-processed image until it meets the requirements;

[0141] Counting the images to be checked, and sorting the images to be checked based on the corresponding acquisition time, obtaining a corresponding check image group, and feeding back to the responsible doctor of the corresponding patient;

[0142] The process of comparing the high-quality image with the original endoscopic image includes:

[0143] The mean and variance of the corresponding high-quality image and the original endoscopic image are obtained respectively, and based on them, an enhancement coefficient between the corresponding high-quality image and the original endoscopic image is obtained; the enhancement coefficient is obtained by the following formula:

[0144]

[0145] Where, Var(F0_I) and Var(F1_I) represent the brightness variance of the original endoscopic image and the high-quality image, respectively; Mean(F0_I) and Mean(F1_I) represent the mean of the contrast pair of the original endoscopic image and the high-quality image, respectively; Used to measure the change ratio of image brightness; Used to measure the change ratio of image contrast;

[0146] μ represents the weight coefficient, which is determined by the staff based on historical work experience;

[0147] A ratio threshold is set. If the enhancement coefficient is not lower than the enhancement threshold, it indicates that the corresponding high-quality image meets the requirements. If the enhancement coefficient is lower than the enhancement threshold, it indicates that the corresponding high-quality image does not meet the requirements.

[0148] It should be further explained that, in the specific implementation process, the coordinate systems corresponding to the pixel point coordinates involved in the present invention are all constructed in advance.

[0149] The present invention significantly improves the quality of endoscopic images through steps such as image preprocessing, feature extraction, feature subgraph processing, feature enhancement and image fusion; in particular, through adaptive grayscale equalization processing and improved wavelet threshold function, the contrast and detail features of the image are enhanced, the noise is reduced, and the image is clearer and more accurate; and the image processing process is simplified.

[0150] Example 2

[0151] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, and an endoscope image processing system is provided, including:

[0152] A data acquisition module, used for acquiring an endoscopic image group and endoscopic data corresponding to the endoscope; the endoscopic image group is composed of a plurality of endoscopic images;

[0153] An image preprocessing module, used to perform image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image;

[0154] Image processing module; used for performing feature extraction on the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs; performing image processing on the obtained feature sub-graphs to obtain corresponding feature enhanced images, and performing image fusion processing based on the feature enhanced images to obtain corresponding high-quality images;

[0155] Image evaluation module: perform image quality evaluation on the obtained high-quality image, obtain the corresponding image evaluation result, and judge whether the high-quality image meets the requirements based on the image evaluation result. If it meets the requirements, the corresponding high-quality image is used as output; if it does not meet the requirements, it is returned to the image processing module;

[0156] The modules are connected to each other via wired and / or wireless means to achieve data transmission between modules.

[0157] Example 3

[0158] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the endoscopic image processing method and system provided above is implemented.

[0159] Since the electronic device introduced in this embodiment is an electronic device used to implement an endoscopic image processing method and system in the embodiment of the present application, based on the endoscopic image processing method and system introduced in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the electronic device used in the endoscopic image processing method and system in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0160] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0161] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An endoscopic image processing method, characterized in that: include: Step 1: obtaining an endoscopic image group and endoscopic data corresponding to the endoscope; the endoscopic image group is composed of a plurality of endoscopic images; Step 2: performing image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image; Step 3: extracting features from the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs; Step 4: performing image processing on the obtained feature sub-image to obtain a corresponding feature enhanced image, and performing image fusion processing based on the feature enhanced image to obtain a corresponding high-quality image; Step 5: Perform image quality assessment on the obtained high-quality image to obtain an image assessment result, and determine whether the high-quality image meets the requirements based on the image assessment result. If it meets the requirements, the corresponding high-quality image is used as output; if it does not meet the requirements, return to step 3; The process of extracting features from the preprocessed image to obtain corresponding feature points, and dividing the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs includes: Importing the preprocessed image as an input image into a pre-built improved feature model to obtain a corresponding model output result, wherein the model output result includes a plurality of output features corresponding to the preprocessed image; Obtaining the pixel position of the output feature in the preprocessed image, and marking the pixel point corresponding to the pixel position as a feature point; Constructing a blank window, taking the feature point as the center point of the blank window, and performing image interception on the preprocessed image based on the feature point to obtain a corresponding feature sub-image; The construction process of the improved feature model is as follows: The network framework of the improved feature model is defined as a ResNet convolutional neural network with an attention mechanism; the improved feature model includes an attention layer and a perception layer; the attention layer includes a channel attention sublayer and a spatial attention sublayer, and both the attention layer and the perception layer are composed of a convolution layer and a pooling layer, and the perception layer also includes a ReLU activation function; Input the obtained input image into the corresponding channel attention sublayer, and the channel attention sublayer performs maximum pooling operation and average pooling operation on the corresponding input image respectively; obtain the corresponding pooling feature map; input the obtained pooling feature map into the perception layer, and the perception layer performs convolution processing on the corresponding pooling feature map respectively to obtain the corresponding convolution feature map; The formula for convolution processing is: R0 = η (W SFE ×I LR +B); where η() represents the ReLU activation function in the corresponding perception layer; W SFE Represents the weight value, I LR represents the input pooling feature map; B represents the bias term; R0 represents the output convolution feature map; The obtained convolution feature map is added bit by bit; and the corresponding weight coefficient Mc is obtained based on the Sigmoid activation function; Based on the weight coefficient Mc, the corresponding input image is bitwise multiplied to obtain the corresponding channel attention feature; Based on the spatial attention sublayer, the corresponding channel attention features are subjected to maximum pooling and average pooling operations to obtain two spatial feature maps, and the two feature maps are concatenated according to the channels to obtain corresponding channel feature maps; After performing a convolution operation on the obtained channel feature map, a weight factor Ms is obtained after passing through a Sigmoid activation function; based on the weight factor Ms, the corresponding channel attention feature is bitwise multiplied to obtain the corresponding output feature; Acquire a historical endoscopic data set, wherein the endoscopic data set includes a plurality of historical endoscopic images and feature labels corresponding to the corresponding endoscopic images; Performing augmentation processing on the historical endoscopic images in the endoscopic data group, and constructing corresponding training data based on the augmented endoscopic data group; The obtained training data is divided into several training batches, and corresponding pre-training weights are assigned to the corresponding training batches; the training batches with pre-training weights are input into the constructed improved feature model for iterative training, and the training is stopped after the model parameters of the corresponding improved feature model converge after P1 consecutive training batches, and the corresponding improved feature model is output; P1 is a natural number; The process of performing image processing on the obtained feature sub-image to obtain a corresponding feature enhanced image, and performing image fusion processing based on the feature enhanced image to obtain a corresponding high-quality image includes: Based on the wavelet threshold algorithm, the obtained feature sub-image is decomposed to obtain the corresponding high-frequency feature image and low-frequency feature image; Performing grayscale conversion on the low-frequency feature image to obtain a corresponding low-frequency grayscale image, and then obtaining the grayscale value corresponding to each pixel in the corresponding low-frequency grayscale image, and constructing a corresponding grayscale histogram based on the grayscale value; Performing adaptive grayscale equalization processing on the obtained low-frequency grayscale image to obtain a corresponding enhanced low-frequency image; Performing image enhancement on the obtained high-frequency feature image based on an improved wavelet threshold function to obtain an enhanced high-frequency image; Perform wavelet reconstruction on the enhanced low-frequency image and the enhanced high-frequency image to obtain the corresponding enhanced feature image; Counting all enhanced feature images, and performing image stitching on the obtained enhanced feature images based on the inverse process of image interception to obtain the corresponding endoscopic enhanced image; Reading endoscopic data corresponding to a corresponding endoscopic image at a corresponding acquisition time; and acquiring a corresponding image mapping correction coefficient based on the endoscopic data; Inputting the obtained endoscopy enhanced image into the constructed improved feature model table to obtain feature points corresponding to the corresponding endoscopy enhanced image, and marking the corresponding feature points as second feature points; At the same time, the corresponding feature point of the corresponding endoscopic image is read and marked as the first feature point; The Euclidean distances between different first feature points and second feature points are obtained respectively, and at the same time, the image similarity of the feature sub-graphs corresponding to the corresponding first feature points and the second feature points is obtained; and based on the similarity, it is determined whether the corresponding first feature points and the second feature points are matching feature points; if they are matching feature points, the corresponding point association relationship is established; if they are not matching feature points, no other operations are performed; Then, based on the obtained point correlation relationship and in combination with the image mapping correction coefficient, point correction is performed on the second characteristic point in the corresponding endoscopic enhanced image; Performing weighted summation of the point-corrected endoscopic enhanced image and the corresponding endoscopic image to obtain a corresponding high-quality image; The corresponding formula for adaptive grayscale balancing is: Wherein, H_f(t) represents the t-th grayscale value after adaptive grayscale equalization in the corresponding grayscale histogram; f(t) represents the t-th grayscale value in the corresponding grayscale histogram; L represents the part of the histogram that exceeds F(t); F(t) represents the limit threshold corresponding to the grayscale value t, and the corresponding limit threshold F(t) = Avg-[a×(Avg×Num(t)-Avg)]; where Avg represents the average grayscale value; a represents the adjustable coefficient, a∈(0,1); Num(t) represents the grayscale level corresponding to the t-th grayscale value in the grayscale histogram; The formula of the improved wavelet threshold function is: Where V_g(V_p(x,y)) represents the enhanced high-frequency image after being processed by the improved wavelet threshold function; V_p(x,y) represents the high-frequency feature image; |V_p(x,y)| represents the absolute value of the wavelet coefficient corresponding to the pixel point (x,y) in the high-frequency feature image; T represents the adaptive wavelet threshold, where Where M and N represent the length and width of the high-frequency feature image, respectively; s and m represent the preset scale factor and adjustment factor.

2. The endoscopic image processing method according to claim 1, characterized in that: The process of obtaining the endoscopic image group and endoscopic data corresponding to the endoscope includes: Setting a data acquisition unit, wherein the data acquisition unit is composed of a data acquisition terminal and an image acquisition terminal; When the data acquisition unit recognizes that the corresponding endoscope recognizes the generation of the video stream, the data acquisition unit collects the endoscopic data and the video stream corresponding to the corresponding endoscope based on the data acquisition terminal and the image acquisition terminal respectively; At the same time, the image acquisition terminal reads the acquired video stream and divides it into a number of continuous single-frame images based on the acquisition time; and compares the acquired single-frame images with the preset screening criteria; if the corresponding single-frame image meets the screening criteria, the corresponding single-frame image is marked as an endoscopic image; if it does not meet the screening criteria, it is discarded; The marked endoscopic images are counted and sorted based on corresponding acquisition times to obtain corresponding endoscopic image groups.

3. The endoscopic image processing method according to claim 2, characterized in that: The process of performing image preprocessing on the obtained endoscopic image to obtain the corresponding preprocessed image includes: Performing an initial image transformation on the obtained endoscopic image, and marking the endoscopic image after the initial image transformation as an initial image; Perform image decomposition on the marked initial image, and obtain first channel images c corresponding to the corresponding RGB color channels respectively; Perform initial image transformation on the first channel image c to obtain the corresponding blurred image MH; Constructing a corresponding pixel feature matrix based on the first channel image c; performing a Gaussian convolution operation on the pixel feature matrix and a pre-selected Gaussian function to obtain a corresponding Gaussian matrix, and obtaining eigenvalues ​​λ1 and λ2 corresponding to the Gaussian matrix; Based on the eigenvalues ​​λ1 and λ2, pixel values ​​of the pixels in the corresponding blurred image MH are adjusted; The pixel-adjusted blurred image MH is inversely transformed to obtain a second channel image corresponding to the corresponding RGB channel; the obtained second channel image is image fused to obtain a corresponding preprocessed image.

4. The endoscopic image processing method according to claim 3, characterized in that: The formula for the initial image transformation is: Where V out Indicates the output image after the initial image transformation is completed, V in represents the input image participating in the initial image transformation, γ is the brightness correction factor; The formula for the corresponding initial image transformation is: In the formula, I c (i, j) represents the pixel value at the pixel point (i, j) in the first channel image c; V MH (i, j) represents the pixel value at the pixel point (i, j) in the corresponding blurred image MH; ε and I cmin Respectively represent the variance and minimum pixel value of the pixel values ​​in the first channel image c; The formula for the inversion is: Where V2_ c (i, j) represents the pixel value at the pixel point (i, j) in the second channel image; I min (i, j) represents the minimum pixel value at pixel point (i, j); V′ MH (i, j) represents the pixel value at the pixel point (i, j) in the blurred image MH after pixel adjustment.

5. The endoscopic image processing method according to claim 4, characterized in that: The process of performing image quality assessment on the obtained high-quality image, obtaining an image assessment result, and judging whether the high-quality image meets the requirements based on the image assessment result includes: The high-quality image is compared with the original endoscopic image to obtain a corresponding comparison result, and based on the comparison result, it is determined whether the corresponding high-quality image meets the requirements. If it does not meet the requirements, it is marked as an image to be inspected; if it does not meet the requirements, the corresponding high-quality image is fed back to step three as a preprocessing image until it meets the requirements; The obtained images to be checked are counted, and the images to be checked are sorted based on the corresponding acquisition time to obtain a corresponding check image group, and the group is fed back to the responsible doctor of the corresponding patient.

6. An endoscopic image processing system, used to implement an endoscopic image processing method according to any one of claims 1 to 5, characterized in that: include: A data acquisition module, used for acquiring an endoscopic image group and endoscopic data corresponding to the endoscope; the endoscopic image group is composed of a plurality of endoscopic images; An image preprocessing module, used to perform image preprocessing on the obtained endoscopic image to obtain a corresponding preprocessed image; Image processing module; Used to perform feature extraction on the preprocessed image to obtain corresponding feature points, and divide the corresponding preprocessed image based on the feature points to obtain corresponding feature sub-graphs; perform image processing on the obtained feature sub-graphs to obtain corresponding feature enhanced images, and perform image fusion processing based on the feature enhanced images to obtain corresponding high-quality images; Image evaluation module; Performing image quality evaluation on the obtained high-quality image to obtain a corresponding image evaluation result, and judging whether the high-quality image meets the requirements based on the image evaluation result, and if so, outputting the corresponding high-quality image; If not, return to the image processing module.

Images