An image processing method and system of an endoscope

Abstract

The application discloses an image processing method and system of an endoscope, and relates to the technical field of image processing. The method comprises the following steps: collecting an RGB image, calculating the centroid of an edge image as an initial distortion center, adding displacement to the initial distortion center, calculating an adjusted distortion center, using a radial distortion model to calculate a distortion radial distance, using polar coordinate transformation to calculate a corrected coordinate, using a Wiener filter formula to calculate a frequency domain real reflection image, using two-dimensional inverse fast Fourier transform to convert the frequency domain real reflection image into a spatial domain to obtain a recovered spatial domain image, using point-by-point addition to calculate the pixel value of an enhanced image, and generating the enhanced image. The radial distortion model is used to calculate the distortion radial distance, and the polar coordinate transformation is used to calculate the corrected coordinate, so that the geometric accuracy and visual authenticity of the image are improved. The point-by-point addition is used to calculate the pixel value of the enhanced image in combination with the gradient field and the enhancement factor, so that the quality and accuracy of the endoscope image are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an endoscope image processing method and system. BACKGROUND

[0002] With the continuous development of endoscope technology, image processing has become a core component of it. Endoscope technology is widely used in medical diagnosis, industrial detection and other fields. The image quality directly affects the accuracy of diagnosis results and the efficiency of detection. In particular, in the medical field, high-definition endoscope images can help doctors better observe lesions and improve the accuracy of treatment.

[0003] The existing endoscope image processing technology still has some deficiencies. The traditional method cannot fully consider the factors such as camera angle, lens distortion and dynamic changes between consecutive frames in the actual acquisition process when processing image distortion, often leading to unsatisfactory correction effect, affecting the accuracy of subsequent image analysis and processing. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides an endoscope image processing method and system, which solves the problem that the traditional method cannot fully consider the factors such as camera angle, lens distortion and dynamic changes between consecutive frames in the actual acquisition process when processing image distortion, often leading to unsatisfactory correction effect, affecting the accuracy of subsequent image analysis and processing.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides an endoscope image processing method, which comprises,

[0008] Collecting an RGB image, using a Harris corner point response function method to calculate a corner point response value, screening a screening threshold greater than the corner point response value as a feature point, calculating the centroid of an edge image as an initial distortion center, adding the displacement to the initial distortion center to calculate an adjusted distortion center, using a radial distortion model to calculate a distortion radial distance, using polar coordinate transformation to calculate a corrected coordinate, using a bilinear interpolation formula to interpolate the corrected coordinate to generate a final corrected image;

[0009] Using a Wiener filter formula to calculate a frequency domain real reflection image, using a two-dimensional inverse fast Fourier transform to convert the frequency domain real reflection image to a spatial domain to obtain a restored spatial domain image, using a point-by-point addition method to calculate the pixel value of an enhanced image to generate an enhanced image, using color space conversion to map the mapping image to the RGB space to obtain an enhanced RGB image;

[0010] extract the feature vector of the enhanced RGB image, construct a convolutional neural network (CNN) model to calculate the class probability, and perform state detection;

[0011] Construct a visual display of the state detection results, and store the collected and analyzed RGB images.

[0012] As a preferred scheme of the image processing method of the endoscope, the collected RGB image is used to calculate the corrected coordinates using polar coordinate transformation, and the final corrected image is generated, including:

[0013] The continuous RGB images are collected by the CCD sensor of the endoscope device using the continuous frame acquisition method,

[0014] The RGB image is converted to grayscale using the weighted average method, the grayscale value is calculated to generate a grayscale image, the Sobel operator is used to calculate the gradients in the horizontal and vertical directions of the grayscale image, the product of the horizontal and vertical gradients is calculated, the Harris response matrix is constructed, the empirical constant is set using the empirical parameter optimization method, and the corner response value is calculated using the Harris corner point response function method;

[0015] The screening threshold is set using the percentage threshold method, and the feature points greater than the screening threshold are selected as the feature points;

[0016] The position of the feature points in the corresponding grayscale image is extracted, the neighborhood window is set using the sliding window method, the neighborhood window is encoded using the local binary pattern, the LBP value is calculated, the Hamming distance between adjacent LBP values is calculated using the Hamming distance, the matching pair with the smallest Hamming distance is selected using the K nearest neighbor matching, and the inter-frame matching relationship is generated;

[0017] The horizontal and vertical gradients of the first frame of the RGB image are extracted, the edge strength is calculated, and the centroid of the edge image is calculated based on the edge strength, which is used as the initial distortion center;

[0018] The displacement of each pair of adjacent frame feature points is calculated based on the inter-frame matching relationship, and the absolute value sum of all displacements is calculated;

[0019] The displacement is added to the initial distortion center to calculate the adjusted distortion center, and the global distortion center is calculated using the arithmetic average method;

[0020] The distance from the spatial coordinates of the frame grayscale image to the global distortion center is calculated using the Euclidean distance formula, denoted as the radial distance, and the mean value of the radial distance is calculated, the angle of the spatial coordinates of the frame grayscale image relative to the center is calculated using the arctangent function;

[0021] The radial distortion coefficient is calculated using the linear approximation method, and the distorted radial distance is calculated based on the radial distance using the radial distortion model.

[0022] The distortion offset is calculated by subtracting the distortion radial distance from the radial distance, and the corrected coordinates are calculated using polar coordinate transformation;

[0023] The position of the corrected coordinates in the corresponding RGB image is extracted, four nearest neighbor pixels are selected using nearest neighbor search, and the corrected coordinates are interpolated using a bilinear interpolation formula to generate the final corrected image.

[0024] As a preferred scheme of the image processing method of the endoscope, the method comprises:

[0025] The final corrected image is converted to grayscale using a weighted average method, and the grayscale value is calculated to generate an enhanced image, the geometric mean value of the four edge pixels of the enhanced image is calculated using a geometric average method, and the illumination distribution is generated using a linear interpolation method.

[0026] The illumination image is calculated using point-by-point division;

[0027] The standard deviation of the Gaussian function is set using the method of moment estimation, and the point spread function is calculated using the two-dimensional Gaussian function formula;

[0028] The illumination image and the point spread function are represented in the frequency domain using two-dimensional fast Fourier transform, the frequency domain conjugate complex of the point spread function is calculated using conjugate complex operation, the frequency domain modulus square of the illumination image is calculated using modulus square operation, the noise power estimation constant is set using fixed threshold method, and the frequency domain real reflection image is calculated using Wiener filter formula.

[0029] The frequency domain real reflection image is converted to the spatial domain using two-dimensional inverse fast Fourier transform to obtain the restored spatial domain image;

[0030] The normalized edge tensor field is convolved using the Prewitt convolution kernel to obtain the horizontal and vertical gradients, and the horizontal and vertical gradients are combined to obtain the gradient field.

[0031] Based on the spatial domain image, the enhancement factor is set using the maximum value function, and the pixel value of the enhanced image is calculated using point-by-point addition to generate the enhanced image.

[0032] The value range of the enhanced image is limited using the threshold clipping method, the pixel value of the enhanced image greater than 255 is defined as 255, and the pixel value of the enhanced image less than 0 is defined as 0 to obtain the clipped enhanced image.

[0033] The clipped enhanced image is nonlinearly mapped using the Sigmoid function, and the mapped image is mapped to the RGB space using color space conversion to obtain the enhanced RGB image.

[0034] As a preferred scheme of the endoscope image processing method, the feature vector of the enhanced RGB image is extracted, including:

[0035] The enhanced RGB image is separated using color channel separation to generate three color channels, and the mean and variance of the three color channels are calculated using statistical analysis method;

[0036] The mean and variance of the three color channels are combined using feature flat splicing method to generate a feature vector.

[0037] As a preferred scheme of the endoscope image processing method, the convolutional neural network CNN model is constructed to calculate the category probability and perform state detection, including:

[0038] Collecting historical RGB images with labels and performing feature extraction to calculate the feature vector and generate a training set;

[0039] The convolutional neural network CNN model includes an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer:

[0040] The training set is used to train the convolutional neural network CNN model, and the cross-entropy loss function and Adam optimizer are used for parameter iteration optimization;

[0041] The feature vector is input into the trained convolutional neural network CNN model to output the category probability, and the classification threshold is set using the empirical rule. When the category probability is greater than the classification threshold, it is judged as an abnormal state, and when the category probability is less than or equal to the classification threshold, it is judged as a normal state.

[0042] As a preferred scheme of the endoscope image processing method, the visualization display of the state detection result is constructed, including:

[0043] A visualization interface is constructed using the front-end framework React.js to display the state detection result;

[0044] Users who pass real-name verification are allowed to view.

[0045] As a preferred scheme of the endoscope image processing method, the RGB image collected and analyzed is stored, including:

[0046] The collected RGB image and the analysis generated detection result are stored in the central database, and security access measures are set. The central database stores the data in the cloud backup, and regularly checks the integrity of the stored data and the backup data. After the detection is completed, the integrity detection record is generated and stored in the central database.

[0047] In a second aspect, the present application provides an image processing system of an endoscope, comprising,

[0048] The collection correction module is configured to collect the RGB image, calculate the corner response value using the Harris corner point response function method, screen the feature points by screening the threshold value greater than the corner response value, calculate the centroid of the edge image as the initial distortion center, add the displacement to the initial distortion center to calculate the adjusted distortion center, calculate the distortion radial distance using the radial distortion model, calculate the corrected coordinates using the polar coordinate transformation, perform interpolation on the corrected coordinates using the bilinear interpolation formula, and generate the final corrected image.

[0049] The image enhancement module is configured to calculate the frequency domain real reflection image using the Wiener filter formula, perform spatial domain conversion on the frequency domain real reflection image using the two-dimensional inverse fast Fourier transform to obtain the recovered spatial domain image, calculate the pixel value of the enhanced image using the point-by-point addition method to generate the enhanced image, map the mapping image to the RGB space using the color space conversion to obtain the enhanced RGB image.

[0050] The feature detection module is configured to extract the feature vector of the enhanced RGB image, construct the convolutional neural network (CNN) model to calculate the category probability, and perform state detection.

[0051] The visualization storage module is configured to construct the visual display of the state detection result and store the RGB image generated by the collection and analysis.

[0052] In a third aspect, the present application provides a computer device, comprising a memory and a processor, and the memory stores a computer program, wherein the computer program is executed by the processor to implement any step of the image processing method of the endoscope according to the first aspect of the present application.

[0053] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by the processor to implement any step of the image processing method of the endoscope according to the first aspect of the present application.

[0054] The present application has the beneficial effects that: the present application collects an RGB image, uses a Harris corner point response function method to calculate a corner point response value, screens a screening threshold value greater than the corner point response value as a feature point, calculates a centroid of an edge image as an initial distortion center, adds a displacement to the initial distortion center to calculate an adjusted distortion center, uses a radial distortion model to calculate a distortion radial distance, uses polar coordinate transformation to calculate a corrected coordinate, uses a bilinear interpolation formula to interpolate the corrected coordinate to generate a final corrected image; uses a Wiener filter formula to calculate a frequency domain real reflection image, uses a two-dimensional inverse fast Fourier transform to convert the frequency domain real reflection image into a spatial domain to obtain a restored spatial domain image, uses a point-by-point addition method to calculate a pixel value of an enhanced image to generate an enhanced image, uses color space conversion to map the mapping image to an RGB space to obtain an enhanced RGB image; improves image geometric accuracy and visual authenticity, and enhances the quality and precision of the endoscope image. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Fig. 1 A flowchart of the image processing method of the endoscope in embodiment 1.

[0057] Fig. 2 A schematic diagram of the image processing system of the endoscope in embodiment 1.

[0058] Fig. 3 A schematic diagram of generating the final corrected image in embodiment 1.

[0059] Fig. 4 A flowchart of the enhanced RGB image in embodiment 1. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings in the specification.

[0061] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0062] Second, the "one embodiment" or "an embodiment" referred to herein can include a particular feature, structure, or characteristic. The various embodiments appearing at different places in the specification are not necessarily all cumulative or alternative embodiments of each other. Moreover, terms such as "first" and "second" are used herein only to describe a certain feature, structure, or characteristic, and do not imply that the features, structures, or characteristics are in any way prioritized, ranked, or otherwise temporally ordered.

[0063] Embodiment 1, Reference Figs. 1 to 4 As a first embodiment of the present application, the embodiment provides an image processing method of an endoscope, comprising the following steps:

[0064] S1, collect RGB images, calculate the corner response value using the Harris corner point response function method, screen the feature points greater than the screening threshold value of the corner response value, calculate the centroid of the edge image as the initial distortion center, add the displacement to the initial distortion center, calculate the adjusted distortion center, calculate the distortion radial distance using the radial distortion model, calculate the corrected coordinates using the polar coordinate transformation, use the bilinear interpolation formula to interpolate the corrected coordinates to generate the final corrected image;

[0065] Specifically, collect RGB images, calculate the corrected coordinates using the polar coordinate transformation to generate the final corrected image, comprising:

[0066] Collect continuous RGB images using the continuous frame acquisition method through the CCD sensor of the endoscope device, pre-process the continuous RGB images, including using blur filtering for denoising processing, and normalizing the RGB images;

[0067] Convert the RGB images to grayscale using the weighted average method, calculate the grayscale value to generate a grayscale image, calculate the gradients of the grayscale image in the horizontal and vertical directions using the Sobel operator, calculate the product of the gradients in the horizontal and vertical directions, construct the Harris response matrix, set the empirical constant using the empirical parameter tuning method, calculate the corner response value using the Harris corner point response function method, the formula is:

[0068] R(x,y)=det(M)-k·trace(M) 2 ,

[0069] Wherein R(x,y) is the corner response value at the spatial coordinates (x,y) of the grayscale image, det(M) is the determinant of the Harris response matrix M, G x is the gradient in the horizontal direction, indicating the intensity change of the image in the horizontal direction, G y is the gradient in the vertical direction, indicating the intensity change of the image in the vertical direction, and trace(M) is the trace of the Harris response matrix M, k is an empirical constant used to balance the influence of determinant and trace;

[0070] A percentage threshold method is used to set a screening threshold, and a feature point greater than the screening threshold of the corner response value is screened;

[0071] The position of the feature point in the corresponding grayscale image is extracted, a neighborhood window is set using a sliding window method, the neighborhood window is encoded using a local binary pattern, the LBP value is calculated, the Hamming distance between adjacent LBP values is calculated using a Hamming distance, and the matching pair with the smallest Hamming distance is selected using a K-nearest neighbor matching method, and an inter-frame matching relationship is generated;

[0072] The horizontal and vertical gradients of the first frame of the RGB image are extracted, and the edge strength is calculated, and the formula is:

[0073]

[0074] where E 1 (x,y) is the edge strength of the first frame of the grayscale image at the spatial coordinates (x,y), is the horizontal gradient of the first frame, is the vertical gradient of the first frame;

[0075] The centroid of the edge image is calculated based on the edge strength, which is used as the initial distortion center, and the formula is:

[0076]

[0077] where and are the initial distortion center coordinates of the first frame of the grayscale image, and x and y are the spatial coordinates of the grayscale image;

[0078] The displacement of each pair of adjacent frame feature points is calculated based on the inter-frame matching relationship, and the sum of the absolute values of all displacements is calculated, and the formula is:

[0079]

[0080] where Δx t and Δy t are the horizontal and vertical displacements of the matching feature points of the tth frame and the t+1th frame, and are the coordinates of the tth frame feature points, i represents the feature point index, and t represents the frame index, and are the coordinates of the matching feature points of the t+1th frame, j represents the feature point index, and t+1 represents the frame index;

[0081] The displacement is added to the initial distortion center, and the adjusted distortion center is calculated, and the global distortion center is calculated using an arithmetic average method;

[0082] The distance of the spatial coordinates of the frame gray image to the global distortion center is calculated using the Euclidean distance formula, denoted as the radial distance, and the mean value of the radial distance is calculated, and the angle of the spatial coordinates of the frame gray image relative to the center is calculated using the inverse tangent function;

[0083] The radial distortion coefficient is calculated using a linear approximation method, and the formula is:

[0084]

[0085] where k1 is the radial distortion coefficient, S is the absolute value sum, N is the total number of matched feature points, r avg is the mean value of the radial distance;

[0086] Based on the radial distance, the distorted radial distance is calculated using the radial distortion model, and the formula is:

[0087]

[0088] where is the distorted radial distance of the t-th frame, r t is the radial distance of the t-th frame;

[0089] The distortion offset is calculated by subtracting the distorted radial distance from the radial distance;

[0090] The corrected coordinates are calculated using the polar coordinate transformation, and the formula is:

[0091] x ′t =x c +(r t +Δr t )cos(θ t ), y ′t =y c +(r t +Δr t )sin(θ t ),

[0092] where x′ t and y′ t are the corrected coordinates, x c and y c are the global distortion center, Δr t is the distortion offset of the t-th frame, and θ t is the angle of the t-th frame;

[0093] The position of the corrected coordinates in the corresponding RGB image is extracted, the four nearest neighbor pixels are selected using the nearest neighbor search, and the corrected coordinates are interpolated using the bilinear interpolation formula to generate the final corrected image.

[0094] The CCD sensor of the endoscope device is used to collect continuous RGB images in real time by using a continuous frame acquisition method, which can improve the efficiency and accuracy of image acquisition; the weighted average method is used for gray scale conversion, which can retain important structural information in the image and reduce noise in the image; the Harris corner point response function method can effectively identify the stable feature points in the image, greatly improving the accuracy and robustness of image matching; by calculating the displacement of the feature points of each pair of adjacent frames, the dynamic change of the image can be efficiently estimated, and the distortion center of the image can be further adjusted accurately; the arithmetic average method is used to calculate the global distortion center, and the radial distance and angle are calculated to further optimize the distortion correction; the radial distortion model is used to accurately correct the image according to the calculated radial distortion distance, so as to eliminate the image distortion caused by the lens distortion; the bilinear interpolation method is used to interpolate the corrected coordinates, which can not only effectively reconstruct the image, but also avoid the possible sawtooth effect in the traditional interpolation method, thereby providing a smoother and higher quality image.

[0095] S2, using the Wiener filter formula to calculate the frequency domain real reflection image, using the two-dimensional inverse fast Fourier transform to convert the frequency domain real reflection image into the spatial domain to obtain the recovered spatial domain image, using the point-by-point addition to calculate the pixel value of the enhanced image to generate the enhanced image, using the color space conversion to map the mapping image to the RGB space to obtain the enhanced RGB image;

[0096] Specifically, the Wiener filter formula is used to calculate the frequency domain real reflection image to obtain the enhanced RGB image, including:

[0097] The final corrected image is converted into a gray scale using the weighted average method, the gray scale value is calculated to generate an enhanced image, the geometric average value of the four edge pixels of the enhanced image is calculated using the geometric average method, and the illumination distribution is generated using the linear interpolation method.

[0098] The illumination image is calculated using the point-by-point division method, and the formula is:

[0099]

[0100] Where L corr (u,v) is the illumination image at the spatial coordinates (u,v), I(u,v) is the enhanced image at the spatial coordinates (u,v), u and v are the spatial coordinates of the enhanced image, and epsilon is a minimum value to avoid division by zero and suppress noise amplification.

[0101] The standard deviation of the Gaussian function is set using the matrix estimation method, and the point spread function is calculated using the two-dimensional Gaussian function formula, which is:

[0102]

[0103] wherein h λ (u,v) is a point spread function at spatial coordinates (u,v), λ is a wavelength of light, and σ is a standard deviation of the Gaussian function;

[0104] The illumination image and the point spread function are represented in the frequency domain using a two-dimensional fast Fourier transform, the frequency domain conjugate complex of the point spread function is calculated using a conjugate complex operation, the frequency domain modulus square of the illumination image is calculated using a modulus square operation, a noise power estimation constant is set using a fixed threshold method, and the frequency domain real reflectance image is calculated using a Wiener filter formula, which is:

[0105]

[0106] wherein O freq (q,p) is a real reflectance image at frequency domain coordinates (q,p), ρ[L corr (u,v)] is a frequency domain modulus square of the illumination image, ρ[h λ ] * is a frequency domain conjugate complex of the point spread function, |ρ[h λ ]| 2 is a modulus square of the point spread function in the frequency domain, and K is a noise power estimation constant;

[0107] The frequency domain real reflectance image is converted into the spatial domain using a two-dimensional inverse fast Fourier transform to obtain a recovered spatial domain image;

[0108] The normalized edge tensor field is convolved using a Prewitt convolution kernel to obtain horizontal and vertical gradients, and the horizontal and vertical gradients are combined to obtain a gradient field;

[0109] Based on the spatial domain image, an enhancement factor is set using a maximum function, and the pixel value of an enhanced image is calculated using a point-by-point addition to generate the enhanced image, which is:

[0110]

[0111] wherein O enh (u,v) is a pixel value of the enhanced image at spatial coordinates (u,v), O(u,v) is the recovered spatial domain image at spatial coordinates (u,v), α is the enhancement factor, is a gradient field at spatial coordinates (u,v);

[0112] The value range of the enhanced image is limited using a threshold clipping method, the pixel value of the enhanced image greater than 255 is defined as 255, and the pixel value of the enhanced image less than 0 is defined as 0 to obtain a clipped enhanced image;

[0113] The enhanced image after cutting is subjected to non-linear gray mapping using a Sigmoid function, and the mapped image is mapped to an RGB space using color space conversion to obtain an enhanced RGB image.

[0114] By performing Fourier transform on the illumination image and the point spread function, the influence of noise in the image can be minimized, thereby improving the clarity and quality of the image. The linear interpolation method for generating the illumination distribution further improves the uniformity of the light in the image, so that the final generated enhanced image presents a smoother and more natural light transition, enhancing the structural details in the image. It is crucial for improving the edge clarity and detail capture in the image. The maximum function is used to set the enhancement factor, which ensures that the enhancement effect of the image is proportional to the overall structure and brightness of the image. The Sigmoid function is used for non-linear gray mapping, which can adjust the contrast and brightness of the image as needed, so that the image is more balanced in vision. Through non-linear gray mapping and color space conversion, the final generated RGB image is more balanced and natural in visual effect, and has a wide application prospect.

[0115] S3, extracting a feature vector of the enhanced RGB image, constructing a convolutional neural network (CNN) model to calculate a class probability, and performing state detection;

[0116] Specifically, the feature vector of the enhanced RGB image is extracted, including:

[0117] The enhanced RGB image is separated using color channel separation to generate three color channels, and the mean and variance of the three color channels are calculated using statistical analysis method.

[0118] The mean and variance of the three color channels are combined using the feature flat splicing method to generate a feature vector.

[0119] By calculating the mean and variance of each channel separately, the basic statistical characteristics of the color channel can be obtained, and the central tendency of the color distribution of the channel, and the variance measures the amplitude of the color change, both of which provide rich information for image features. By feature splicing method, the mean and variance values of three color channels are combined into a unified feature vector, which can maintain the unique information of each channel while retaining the overall image color distribution characteristics. Compared with directly using the original image or large-scale pixel data, this feature vector not only improves the processing efficiency, but also reduces the computational complexity. Especially in image recognition, target detection and image classification applications, this feature vector can efficiently provide the core information of the image, help the algorithm make fast and accurate decisions, and fuse the information of different color channels into a compact feature vector, which helps to improve the effect of machine learning model in image classification, recognition and other tasks. In the machine learning model training process, using these extracted feature vectors as data input can not only improve the training speed, but also enhance the sensitivity of the model to color patterns in the image.

[0120] Further, a convolutional neural network CNN model is constructed to calculate the category probability and perform state detection, including:

[0121] Collecting historical RGB images with labels and performing feature extraction, calculating feature vectors, and generating a training set;

[0122] The convolutional neural network CNN model includes an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer:

[0123] The training set is used to train the convolutional neural network CNN model, and the cross-entropy loss function and Adam optimizer are used for parameter iteration optimization;

[0124] The feature vector is input into the trained convolutional neural network CNN model to output the category probability, and the experience rule is used to set the classification threshold. When the category probability is greater than the classification threshold, it is judged as an abnormal state, and when the category probability is less than or equal to the classification threshold, it is judged as a normal state.

[0125] By extracting representative features from images, the complexity of the problem can be effectively reduced, the classification accuracy can be improved, and a convolutional neural network (CNN) model can be constructed, including an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer. The convolutional layer is responsible for extracting local features from images, the pooling layer preserves the most important information by reducing the dimensionality of the feature map, the activation function introduces nonlinearity to increase the expressive power of the model, and the fully connected layer maps the extracted features to the final class output. By using the cross-entropy loss function and the Adam optimizer, the convolutional neural network can automatically adjust the parameters to achieve more accurate classification capabilities. The update process can continuously iterate and optimize the weights of the network, so that the model can learn the correct patterns from the sample data in the training set, improve the classification ability and generalization ability of the model, and the convolutional neural network can automatically recognize complex patterns in images through multi-level feature extraction and deep learning, improving the accuracy and robustness of classification.

[0126] S4, constructing a visual display of the state detection result, storing the collected and analyzed RGB images;

[0127] Specifically, constructing a visual display of the state detection result includes:

[0128] Using the front-end framework React.js to build a visual interface to display the state detection result;

[0129] Allowing users who have passed real-name verification to view.

[0130] React.js's virtual DOM can improve page rendering efficiency and reduce unnecessary DOM operations, thereby improving user experience, especially in real-time data update scenarios, ensuring smooth page response. Through React.js, the front-end interface can quickly respond to changes in back-end data, realize real-time display and update of state detection results, and through strict identity verification and access control, ensure that each user has clear identity authentication when performing viewing operations, reducing the risk of data leakage. Through real-time and intuitive visual display, users can identify abnormal states in the system in the first time, thereby making timely decisions.

[0131] Further, storing the collected and analyzed RGB images includes:

[0132] The collected RGB images and the analysis generated detection results are stored in the central database, and security access measures are set. The central database stores the data in the cloud backup, and periodically detects the integrity of the stored data and the backup data. After the detection is completed, the integrity detection record is generated and stored in the central database.

[0133] The system can save relevant data in the central database by collecting RGB images and analyzing to generate detection results. Security access measures including identity authentication, access control, encrypted communication, etc. are crucial in the medical industry, financial industry and government data processing. It not only improves the security of data storage, but also ensures compliance with relevant laws and industry standards. Cloud backup can provide disaster recovery capability in case of local hardware damage or natural disasters, ensuring data is not lost, providing high availability, and integrity detection technology can compare the original state and stored state of the data by calculating hash values, checksums, etc. to ensure data accuracy. By generating integrity detection records and synchronously storing them to the central database, detailed log information can be provided for subsequent audit, fault analysis and optimization.

[0134] The embodiment also provides an image processing system of an endoscope, comprising:

[0135] The collection correction module is configured to collect RGB images, calculate corner response values using a Harris corner point response function method, screen feature points greater than a screening threshold of the corner response values, calculate a centroid of an edge image as an initial distortion center, add a displacement to the initial distortion center to calculate an adjusted distortion center, calculate a distortion radial distance using a radial distortion model, calculate corrected coordinates using a polar coordinate transformation, interpolate the corrected coordinates using a bilinear interpolation formula, and generate a final corrected image;

[0136] The image enhancement module is configured to calculate a frequency domain real reflection image using a Wiener filter formula, perform spatial domain conversion on the frequency domain real reflection image using a two-dimensional inverse fast Fourier transform to obtain a restored spatial domain image, calculate pixel values of an enhanced image using point-by-point addition, generate an enhanced image, map the enhanced image to an RGB space using color space conversion to obtain an enhanced RGB image;

[0137] The feature detection module is configured to extract feature vectors of the enhanced RGB image, construct a convolutional neural network (CNN) model to calculate class probabilities, and perform state detection.

[0138] The visualization storage module is configured to construct a visual display of the state detection results and store the collected and analyzed RGB images.

[0139] The embodiment also provides a computer device suitable for the image processing method of the endoscope, comprising a memory and a processor. The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the image processing method of the endoscope as described in the above embodiments.

[0140] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, a carrier network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, a trackball or a touchpad arranged on the shell of the computer device, or an external keyboard, a touchpad or a mouse, etc.

[0141] The embodiment also provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the image processing method of the endoscope. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0142] To sum up, the application collects the RGB image, uses the Harris corner point response function method to calculate the corner point response value, screens the screening threshold value greater than the corner point response value as the feature point, calculates the centroid of the edge image as the initial distortion center, adds the displacement to the initial distortion center to calculate the adjusted distortion center, uses the radial distortion model to calculate the distortion radial distance, uses the polar coordinate transformation to calculate the corrected coordinates, uses the bilinear interpolation formula to interpolate the corrected coordinates to generate the final corrected image, uses the Wiener filtering formula to calculate the frequency domain real reflection image, uses the two-dimensional inverse fast Fourier transform to convert the frequency domain real reflection image to the spatial domain to obtain the restored spatial domain image, uses the point-by-point addition method to calculate the pixel value of the enhanced image to generate the enhanced image, uses the color space conversion to map the mapping image to the RGB space to obtain the enhanced RGB image, improves the image geometric accuracy and visual reality, and enhances the quality and precision of the endoscope image.

[0143] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. An image processing method of an endoscope, characterized by: The method comprises the following steps: The RGB image is collected, the corner response value is calculated by using the Harris corner point response function method, the feature points are selected by screening the threshold value greater than the corner response value, the centroid of the edge image is calculated as the initial distortion center, the displacement is added to the initial distortion center, the adjusted distortion center is calculated, the radial distortion distance is calculated by using the radial distortion model, the corrected coordinates are calculated by using the polar coordinate transformation, the bilinear interpolation formula is used for interpolation on the corrected coordinates, and the final corrected image is generated; The frequency domain real reflection image is calculated by using the Wiener filter formula, the spatial domain conversion is performed on the frequency domain real reflection image by using the two-dimensional inverse fast Fourier transform to obtain the restored spatial domain image, the pixel value of the enhanced image is calculated by using the point-by-point addition method to generate the enhanced image, and the mapping image is mapped to the RGB space by using the color space conversion to obtain the enhanced RGB image; The feature vector of the enhanced RGB image is extracted, the convolutional neural network (CNN) model is constructed to calculate the category probability, and the state detection is performed; The state detection result is visually displayed, and the collected and analyzed RGB images are stored; The RGB image is collected, the polar coordinate transformation is used to calculate the corrected coordinates, and the final corrected image is generated, which comprises the following steps: The continuous RGB images are collected by using the continuous frame acquisition method through the CCD sensor of the endoscope device, The RGB image is converted into a grayscale image by using the weighted average method, the grayscale value is calculated, the Sobel operator is used to calculate the gradients in the horizontal and vertical directions of the grayscale image, the product of the gradients in the horizontal and vertical directions is calculated, the Harris response matrix is constructed, the empirical constant is set by using the empirical parameter optimization method, and the corner response value is calculated by using the Harris corner point response function method; The feature points are selected by screening the threshold value greater than the corner response value; The positions of the feature points in the corresponding grayscale image are extracted, the neighborhood window is set by using the sliding window method, the neighborhood window is encoded by using the local binary pattern, the LBP value is calculated, the Hamming distance between adjacent LBP values is calculated by using the Hamming distance, and the matching pair with the minimum Hamming distance is selected by using the K nearest neighbor matching to generate the inter-frame matching relationship; The gradients in the horizontal and vertical directions of the first frame of the RGB image are extracted, the edge strength is calculated, and the centroid of the edge image is calculated as the initial distortion center based on the edge strength; The displacement of each pair of adjacent frame feature points is calculated based on the inter-frame matching relationship, and the absolute value sum of all displacements is calculated; The displacement is added to the initial distortion center to calculate the adjusted distortion center, and the global distortion center is calculated by using the arithmetic average method; The distance from the spatial coordinates of the frame grayscale image to the global distortion center is calculated by using the Euclidean distance formula, which is recorded as the radial distance, and the mean value of the radial distance is calculated, and the angle of the spatial coordinates of the frame grayscale image relative to the center is calculated by using the arctangent function; The radial distortion coefficient is calculated by using the linear approximation method, and the radial distortion distance is calculated based on the radial distance by using the radial distortion model; The distortion offset is calculated by subtracting the radial distortion distance from the radial distance, and the corrected coordinates are calculated by using the polar coordinate transformation. The corrected coordinates are extracted in the corresponding RGB image position, four nearest neighbor pixels are selected using nearest neighbor search, the corrected coordinates are interpolated using bilinear interpolation formula, and the final corrected image is generated; The calculation of the frequency domain real reflection image using the Wiener filter formula obtains the enhanced RGB image, including: The final corrected image is converted to grayscale using the weighted average method, the grayscale value is calculated to generate an enhanced image, the geometric mean of the four edge pixels of the enhanced image is calculated using the geometric average method, and the illumination distribution is generated using the linear interpolation method; The illumination image is calculated using point-by-point division; The standard deviation of the Gaussian function is set using the moment estimation method, and the point spread function is calculated using the two-dimensional Gaussian function formula; The illumination image and the point spread function are represented in the frequency domain using two-dimensional fast Fourier transform, the frequency domain conjugate complex of the point spread function is calculated using conjugate complex operation, the frequency domain modulus square of the illumination image is calculated using modulus square operation, the noise power estimation constant is set using fixed threshold method, and the frequency domain real reflection image is calculated using the Wiener filter formula; The spatial domain image is converted from the frequency domain real reflection image using two-dimensional inverse fast Fourier transform to obtain the restored spatial domain image; The normalized edge tensor field is convolved using the Prewitt convolution kernel to obtain the horizontal and vertical gradients, and the horizontal and vertical gradients are combined to obtain the gradient field; Based on the spatial domain image, the enhancement factor is set using the maximum value function, and the pixel value of the enhanced image is calculated using point-by-point addition to generate the enhanced image; The value range of the enhanced image is limited using the threshold clipping method, the pixel value of the enhanced image greater than 255 is defined as 255, and the pixel value of the enhanced image less than 0 is defined as 0 to obtain the clipped enhanced image; The clipped enhanced image is nonlinearly mapped using the Sigmoid function, and the mapped image is mapped to the RGB space using color space conversion to obtain the enhanced RGB image.

2. The image processing method of an endoscope according to claim 1, characterized by: The feature vector of the enhanced RGB image is extracted, including: The enhanced RGB image is separated using color channel separation to generate three color channels, and the mean and variance of the three color channels are calculated using statistical analysis; The mean and variance of the three color channels are combined to generate the feature vector using the feature flat splicing method.

3. The image processing method of an endoscope according to claim 2, characterized by: The convolutional neural network (CNN) model is constructed to calculate the category probability and perform state detection, including: Collecting historical RGB images with labels and performing feature extraction, calculating feature vectors, and generating a training set; The convolutional neural network (CNN) model is constructed, including an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer, and an output layer: The convolutional neural network (CNN) model is trained using the training set, and the parameters are iteratively optimized using the cross-entropy loss function and the Adam optimizer; The feature vector is input into the trained convolutional neural network (CNN) model to output the category probability, and the classification threshold is set using the empirical rule. When the category probability is greater than the classification threshold, it is judged as an abnormal state, and when the category probability is less than or equal to the classification threshold, it is judged as a normal state.

4. The image processing method of an endoscope according to claim 3, characterized by: The state detection result is visualized and displayed, including: The visual interface is built using the front-end framework React.js, and the state detection results are displayed. Allow users to access through real-name verification.

5. The image processing method of an endoscope according to claim 4, characterized by: The RGB images collected and analyzed are stored, including: The collected RGB images and the analysis generated detection results are stored in the central database, and security access measures are set. The central database will store the data in the cloud backup, and regularly check the integrity of the stored data and backup data. After the detection is completed, the integrity detection record is generated and stored in the central database.

6. An image processing system of an endoscope based on the image processing method of any one of claims 1 to 5, characterized by: Including, The collection correction module is used to collect RGB images, calculate the corner response value using the Harris corner point response function method, filter the threshold value greater than the corner response value as the feature point, calculate the centroid of the edge image as the initial distortion center, add the displacement to the initial distortion center, calculate the adjusted distortion center, calculate the distortion radial distance using the radial distortion model, calculate the corrected coordinates using the polar coordinate transformation, and generate the final corrected image using the bilinear interpolation formula for interpolation of the corrected coordinates; The image enhancement module is used to calculate the frequency domain real reflection image using the Wiener filter formula, convert the frequency domain real reflection image to the spatial domain using the two-dimensional inverse fast Fourier transform, obtain the restored spatial domain image, calculate the pixel value of the enhanced image using the point-by-point addition method, generate the enhanced image, map the mapping image to the RGB space using the color space conversion, and obtain the enhanced RGB image; The feature detection module is used to extract the feature vector of the enhanced RGB image, construct a convolutional neural network CNN model to calculate the class probability, and perform state detection. The visualization storage module is used to build a visual display of the state detection results, and store the RGB images collected and analyzed.

7. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: The processor executes the computer program to realize the steps of the endoscope image processing method of any one of claims 1-5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the endoscope image processing method of any one of claims 1-5.

Images