Tympanic membrane puncture assisted navigation system based on image processing and machine learning

Abstract

The present invention relates to an auxiliary navigation system for tympanic membrane puncture based on image processing and machine learning. An image segmentation module uses a convolutional neural network to segment the ear canal and eardrum images; a feature extraction module uses a deep learning model to extract key feature points from the segmented image, including the morphological features, position and surrounding structures of the eardrum; a path planning module collects surgical videos and puncture path data and uses a reinforcement learning algorithm to train a path planning model, so that the path planning module generates an optimal puncture path in the image; a real-time path adjustment module uses real-time image processing technology to update the images of the ear canal and eardrum, dynamically adjusts the puncture path through the path planning model, and controls the accuracy of the path during the operation; a navigation interface module develops a surgical navigation interface, displays high-definition images, puncture paths and related anatomical structures in real time, and integrates 3D views to provide multi-angle observation of surgical scenes.

Description

Technical Field

[0001] The present invention relates to a tympanic membrane puncture auxiliary navigation system, and in particular to a tympanic membrane puncture auxiliary navigation system based on image processing and machine learning. Background Art

[0002] The shortcomings and drawbacks of the current assisted navigation system for tympanic membrane puncture are mainly concentrated in several aspects: accuracy, real-time, stability, user-friendliness and adaptability. First, in terms of accuracy, the existing system still has certain errors in positioning and guiding the puncture needle. This is mainly due to the limitations of image processing algorithms and sensor technology. Although high-resolution endoscopic cameras can provide high-definition images, in actual operation, due to factors such as image noise, illumination changes, and complex anatomical structures in the ear canal, the accuracy of image segmentation and feature extraction will be affected, thereby affecting the accuracy of the puncture path. In addition, the existing image processing algorithm is still insufficient in dealing with complex backgrounds and tiny features, especially when dealing with subtle changes and movements of the mucosa in the ear canal, which is prone to misjudgment and missed judgment. In terms of real-time, although the existing system can provide certain real-time navigation functions, the processing speed and response time still cannot fully meet the high requirements of clinical surgery. Especially during the operation, any delay or jamming may affect the doctor's operation fluency and surgical safety. The current system has the problems of large computational complexity and slow processing speed in the entire process from image acquisition, processing, feature extraction to path planning and feedback control. In addition, the system's hardware configuration and software optimization also directly affect real-time performance. On some low-configuration devices, the processing speed often cannot achieve the ideal effect, resulting in a slow system response and failure to provide navigation suggestions in a timely manner. Stability is also a major problem with existing systems. Due to the complex environment of ear canal surgery, the system will be subject to various interferences during operation, including slight head movements of the patient and vibrations caused by breathing. These factors will affect the stability of image acquisition and, in turn, the reliability of subsequent processing. The existing system needs to be improved in terms of anti-interference ability and adaptive adjustment. The system is easily affected by environmental noise and changes in lighting during processing, resulting in unstable image quality, which in turn affects navigation accuracy.

[0003] In terms of user-friendliness, the existing systems still have deficiencies in interface design and operation procedures. During surgery, doctors need to quickly and accurately obtain navigation information, but complex operation interfaces and cumbersome operation steps often increase the burden on doctors. The interface design of some systems is not intuitive enough, and the information display is not clear enough, which causes doctors to spend extra time to understand and operate the system during surgery. In addition, the existing systems are not interactive and intelligent enough in surgical operations, and cannot be personalized according to the doctor's operating habits and actual needs, which affects the efficiency of surgery. In terms of adaptability, the adaptability of existing systems in dealing with different patients and different surgical scenarios is not strong enough. The anatomical structure involved in ear canal surgery is complex and changeable, and the ear canal shape, size, and position of different patients are different. The existing system is not good enough in adapting to these differences. Especially for some complex cases, including narrow ear canals and abnormal tympanic membranes, the existing system is prone to errors in navigation and positioning. In addition, the system's ability to respond to emergencies and abnormal situations is also limited, and it lacks intelligent adjustment mechanisms and flexible processing strategies. Summary of the invention

[0004] The purpose of the present invention is to provide a tympanic membrane puncture auxiliary navigation system based on image processing and machine learning, so as to solve some of the disadvantages and shortcomings pointed out in the background technology.

[0005] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: comprising an image segmentation module, which uses a convolutional neural network to segment the ear canal and tympanic membrane images to separate the tympanic membrane and its surrounding anatomical structures;

[0006] The convolutional neural network architecture adopts the U-Net network architecture, the encoder part extracts the high-dimensional features of the image layer by layer, and the decoder part gradually restores the spatial information of the image through jump connections;

[0007] Then, the cross entropy loss function and Dice coefficient are used as loss functions to balance the accuracy of the segmentation results and the area overlap. The Adam optimizer is used to optimize the model parameters and the learning rate decay strategy is used to prevent overfitting.

[0008] The feature extraction module uses a deep learning model to extract key feature points from the segmented image, including the morphological features, location, and surrounding structures of the tympanic membrane;

[0009] A path planning module collects surgical videos and puncture path data and uses a reinforcement learning algorithm to train a path planning model so that the path planning module generates an optimal puncture path in the image;

[0010] The real-time path adjustment module uses real-time image processing technology to update the images of the ear canal and tympanic membrane, dynamically adjusts the puncture path through the path planning model, and controls the accuracy of the path during the operation;

[0011] Navigation interface module, develops a surgical navigation interface, displays high-definition images, puncture paths and related anatomical structures in real time, and integrates 3D views to provide multi-angle observation of surgical scenes.

[0012] Furthermore, the convolutional neural network architecture design process is as follows:

[0013] S1, using the U-Net network architecture to extract and segment the image features. The encoder part extracts the high-dimensional features of the image layer by layer, using the convolution operation formula:

[0014] C(x,k)=∫ Ω σ(W(k)·x(tk)+b(k))dk

[0015] Where x represents the input image data, W(k) and b(k) are position-dependent learning parameters, σ is the activation function, t is the variable in the convolution operation, and Ω is the entire input domain;

[0016] S2. Custom feature depth adjustment function:

[0017] F(d,z)=α·d 2 ·e -λ·z

[0018] Where d represents the depth of the network, z is the complexity evaluation of the input features, and α and λ are adjustment coefficients to optimize the feature extraction depth of each layer and adapt to different input complexities;

[0019] S3, the decoder part gradually restores the spatial information of the image through upsampling and convolutional layers, and uses jump connections to enhance the retention of structural details. The corresponding spatial information recovery formula is:

[0020] R(y)=∫ s β·y (γ·s) ds

[0021] Where y represents the feature map passed from the encoder to the decoder, β and γ are adjustment coefficients that control the fineness of recovery and the degree of information retention, and S is the spatial range of feature recovery.

[0022] Furthermore, the accuracy of the balanced segmentation result and the area overlap are achieved by combining the cross entropy loss and the Dice coefficient, which is specifically expressed as the loss function:

[0023]

[0024] where y iis the label value, p i is the model prediction, and λ is the weight coefficient used to balance the two terms;

[0025] The Adam optimizer is used to adjust the model weights, where the model weight update rule is:

[0026]

[0027] η t is the learning rate at time t, and are the bias-corrected first-order and second-order moment estimates at time t, ∈ is a small amount to prevent division by zero;

[0028] And implement the learning rate decay strategy, specifically:

[0029] η t+1 =η t ·e -αt

[0030] η t is the learning rate at time t, and α is the decay rate.

[0031] Furthermore, the feature extraction module includes: firstly adopting a model with a residual network structure and operating through a feature extraction formula:

[0032]

[0033] Where x represents the input image data, W l and b l are the weights and biases of the lth layer, α l is the adjustment coefficient of each layer, L is the number of layers in the network, and * represent gradient and convolution operations respectively;

[0034] Then implement the multi-level feature fusion technology through the feature fusion function:

[0035]

[0036] where y i is the feature map output by the i-th layer, β i ,γ i , and δ i It is to integrate adjustment parameters to enhance the model’s adaptability to different anatomical structure sizes and shapes;

[0037] The feature vector is post-processed using the optimization function:

[0038]

[0039] where z jis the component of the feature vector after dimensionality reduction, θ j is the optimization weight, k j is a nonlinear adjustment factor.

[0040] Furthermore, the path planning module construction method includes: using a data collection and preprocessing module, applying a custom image transformation and enhancement algorithm, expressed as:

[0041]

[0042] Where I(x,y) represents the input image frame, represents the gradient of the image at point (x, y), κ is the parameter for adjusting the local contrast sensitivity of the image, and Ω is the domain of the image;

[0043] The design and training module of the reinforcement learning model uses a complex strategy update formula:

[0044]

[0045] where π old and π new They represent the strategies before and after the update, α is the learning rate, γ is the discount factor that affects the current value of future rewards, and r t is the reward obtained at time t, represents the gradient of the policy.

[0046] Furthermore, the path planning module construction process is as follows:

[0047] S1. Use a high-resolution endoscopic camera to capture images of the ear canal and tympanic membrane in real time and apply real-time image processing techniques, including edge enhancement, contrast adjustment, dynamic noise filtering, and illumination compensation;

[0048] S2, using deep learning networks and reinforcement learning algorithms to dynamically generate and adjust the puncture path based on real-time updated image data, and represent new image information and surgical dynamics based on the original path planning;

[0049] S3, integrated real-time feedback system, directly controls surgical tools through real-time tracking and deviation analysis of puncture needle position.

[0050] Furthermore, the real-time capture of the images of the ear canal and the eardrum uses an image processing algorithm:

[0051]

[0052] Where I(ξ,η) represents the intensity of the image at the coordinate (ξ,η), and σ is the Gaussian smoothing parameter used for edge enhancement and contrast adjustment;

[0053] And implement dynamic noise filtering and illumination compensation algorithms:

[0054]

[0055] Where I(t) is the image data at time t, μ(t) is the average brightness of the image at time t, and γ(t) is the dynamic compensation coefficient adjusted according to the change of ambient light.

[0056] Furthermore, the implementation process of the deep learning network and reinforcement learning algorithm is as follows: first, the deep learning network is used to process and analyze the real-time captured ear canal and tympanic membrane images, and a custom feature extraction formula is applied:

[0057]

[0058] Among them I i is the i-th pixel in the image, represents the Laplace operator of the pixel, w i is the weight factor, σ is the standard deviation of the adjusted response intensity;

[0059] Then, the reinforcement learning algorithm is combined to dynamically generate and adjust the puncture path, using the dynamic path optimization formula:

[0060]

[0061] where π old and π new represents the old strategy and the new strategy respectively, s and s′ are the current and next states, a is the action taken, P(s′|s,a) is the transition probability from state s to state s′ through action a, R(s,a,s′) is the reward function, γ is the discount factor for future rewards, Q(s,a) is the action value function, and η is the learning rate.

[0062] Furthermore, the integrated real-time feedback system construction process is as follows:

[0063] S1. Use real-time tracking technology, which is specifically achieved by capturing images with a high-resolution endoscope camera and applying image processing algorithms:

[0064]

[0065] Where I(x,y) is the image brightness function, is the Laplacian operator of the image at (x, y), which is used to detect edges and other high-frequency details, τ is the preset threshold, σ is the parameter for adjusting detail sensitivity, and Ω is the domain of the image;

[0066] S2. Combined with dynamic deviation analysis technology, the specific algorithm is:

[0067] Δ(s)=∫ s ρ(s,a)(κ(s,a,s′)-ν(s,a))ds′

[0068] Where ρ(s,a) is the probability density function of state s and action a, κ(s,a,s′) is the expected reward when achieving state s′, ν(s,a) is the deviation cost function between the expected path and the actual path, and S represents all state spaces;

[0069] S3, and integrated control system, through the feedback control formula:

[0070]

[0071] Adjust the actions of the surgical tool, where λ is the learning rate, π(s,a) represents the policy of taking action a in state s, γ is the discount factor, and Q(s,a) is the value function of action a in state s.

[0072] Beneficial effects of the present invention:

[0073] First, through high-resolution endoscopic cameras and advanced image processing algorithms, the system can provide clear images of the ear canal and tympanic membrane, accurately identify the tympanic membrane and its surrounding structures, and reduce errors in doctor operations. Combined with machine learning algorithms, the system can dynamically learn and optimize the puncture path to improve puncture accuracy.

[0074] Second, the system integrates efficient image processing and machine learning models, which can quickly process and analyze real-time captured image data, and provide instant surgical navigation and path adjustment suggestions. Efficient computing and response speed ensure that doctors can receive timely guidance and feedback during surgery, avoiding delays that affect surgical results.

[0075] Third, the system uses advanced image processing technologies, such as dynamic noise filtering and illumination compensation, to maintain stable image quality in complex surgical environments. Even with interference factors such as slight head movement or illumination changes, the system can still provide stable and reliable images and navigation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flow chart of the tympanic membrane puncture auxiliary navigation system based on image processing and machine learning of the present invention.

[0077] Figure 2 The module flow chart of the convolutional neural network architecture design of the present invention.

[0078] Figure 3 This is a schematic diagram of the operation of the feature extraction module of the present invention.

[0079] Figure 4This is a flow chart of the path planning module of the present invention. DETAILED DESCRIPTION

[0080] The specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0081] Combined with Figure 1 The tympanic membrane puncture assisted navigation system based on image processing and machine learning includes: the system's built-in image segmentation module uses a convolutional neural network with a U-Net architecture to capture and analyze complex biological structures. Its encoder part can deeply extract high-dimensional features of the image, while the decoder restores the spatial information of the image through jump connections, thereby reproducing the image of the tympanic membrane and its surrounding structures. This segmentation process is optimized through cross entropy loss and Dice coefficient to ensure the accuracy of segmentation and the maximum overlap between regions. At the same time, the Adam optimizer is used for parameter adjustment and learning rate decay strategy to prevent model overfitting.

[0082] The feature extraction module further processes these segmented images and uses deep learning algorithms to extract key structural feature points, including the shape and position of the tympanic membrane and specific information about the sensitive areas around it. These data are crucial for determining the optimal location for puncture. The path planning module uses the collected surgical videos and actual puncture path data to train a reinforcement learning algorithm to obtain a model that dynamically generates and adjusts the optimal puncture path during surgery. The model reflects new image information and surgical dynamics in real time, thereby adjusting the puncture strategy according to actual conditions during surgery.

[0083] The real-time path adjustment module is a key part of the system. It integrates real-time image processing technology and can continuously update the image data of the ear canal and tympanic membrane. The system dynamically adjusts the path of the puncture needle through these updated data to ensure the accuracy of the path throughout the operation. Finally, the navigation interface module provides the surgical team with an intuitive operation interface that displays high-definition images, the current puncture path and its corresponding anatomical structure in real time.

[0084] Embodiment 1:

[0085] Combined with Figure 2 The embodiment adopts the U-Net network architecture to extract and segment the image features. The encoder part extracts the high-dimensional features of the image layer by layer, and adopts the convolution operation formula:

[0086] C(x,k)=∫ Ω σ(W(k)·x(tk)+b(k))dk

[0087] Where x represents the input image data, W(k) and b(k) are position-dependent learning parameters, σ is the activation function, t is the variable in the convolution operation, and Ω is the entire input domain.

[0088] Process a 256x256 pixel ear canal and eardrum image with a convolution kernel size of 3x3. First, define the initial value range of the convolution kernel weight W(k) and bias b(k) to be [-0.01, 0.01]. Set the initial weight W(k) to 0.005 and the bias b(k) to 0.001. Use ReLU as the activation function σ(x) = max(0, x). Set a region of the input image x to contain the following pixel values: x = [150, 160, 170, 180, 190, 200, 210, 220, 230]. Perform the convolution operation, first calculate the weighted sum and add the bias:

[0089] W(k)·x(tk)+b(k)

[0090] =0.005·150+0.005·160+0.005·170+0.005·180+0.005·190+0.005·200+0.005·210+0.005·220+0.005·230+0.001

[0091] =1.25+0.80+0.85+0.90+0.95+1.00+1.05+1.10+1.15+0.001

[0092] =9.101

[0093] Then apply the ReLU activation function:

[0094] σ(9.101)=max(0,9.101)=9.101

[0095] This value is part of the output feature map after the convolution operation. Repeat the above steps to traverse the entire image, after multiple layers of convolution and pooling operations, the encoder finally outputs a high-dimensional feature map of size 8x8.

[0096] The embodiment further introduces a custom feature depth adjustment function to optimize the feature extraction depth of each layer and adapt to different input complexities. The specific function is:

[0097] F(d,z)=α·d 2 ·e -λ·z

[0098] Where d represents the depth of the network, z is the complexity estimate of the input features, and α and λ are adjustment coefficients.

[0099] The value range of α is set to 0.1 to 1.0, which is used to control the influence strength of feature extraction depth;

[0100] The value range of λ is set to 0.01 to 0.1 to adjust the impact of input feature complexity.

[0101] After the input image is preprocessed, the system evaluates its complexity and obtains z = 0.5. In the 5th layer of the network (ie, d = 5), α = 0.5 and λ = 0.05 are set.

[0102] First, calculate the feature depth adjustment function:

[0103] F(5,0.5)=0.5·5 2 ·e -0.05·0.5

[0104] Calculate the value at each step:

[0105] 5 2 =25

[0106] 0.05 0.5 = 0.025

[0107] e -0.025 ≈0.975

[0108] Substitute into the calculation:

[0109] F(5,0.5)=0.5·25·0.975=12.1875

[0110] The result of the feature depth adjustment function F(d,z) is 12.1875. In the 5th layer of the network, the feature extraction depth of this layer will be adjusted to 12.1875 times to ensure that this layer can adapt to the complexity of the input image.

[0111] The last embodiment uses the decoder to gradually restore the spatial information of the image. This process is achieved through a combination of upsampling and convolutional layers. At the same time, skip connections are used to enhance the retention of structural details. The spatial information recovery formula is defined as:

[0112] R(y)=∫ s β·y (γ·s) ds

[0113] Where y represents the feature map passed from the encoder to the decoder, β and γ are adjustment coefficients that control the fineness of recovery and the degree of information retention, and S is the spatial range of feature recovery.

[0114] The value range of β is set to 0.5 to 1.5, which is used to control the degree of enlargement or reduction of the feature map;

[0115] The value range of γ is set to 0.1 to 1.0, which is used to adjust the recovery accuracy of feature information.

[0116] The encoder extracts features from an ear canal and eardrum image of 256x256 pixels to obtain a 32x32 pixel feature map. The decoder performs upsampling and convolution operations, while using skip connections to preserve structural details. β = 1.0 and γ = 0.5.

[0117] Calculate the feature recovery of the decoder part:

[0118] R(y)=∫ s 1.0·y (0.5·s) ds

[0119] Set the value of a certain area in the feature map y to y = [1.2, 0.8, 1.5, 1.1], and the spatial range S of feature recovery is [0, 1].

[0120] Calculate the recovery result of each eigenvalue one by one:

[0121] When y=1.2, the feature recovery calculation is 1.2 (0.5·1) =1.095

[0122] When y=0.8, the feature recovery calculation is 0.8 (0.5·1) =0.894

[0123] When y=1.5, the feature recovery calculation is 1.5 (0.5·1) =1.225

[0124] When y=1.1, the feature recovery calculation is 1.1 (0.5·1) =1.048

[0125] Through the above calculations, the restored feature map is:

[0126] R(y)=[1.095,0.894,1.225,1.048]

[0127] Embodiment 2:

[0128] In order to ensure the accuracy of the segmentation results and the area overlap, a combination of cross entropy loss and Dice coefficient is used as the loss function. The specific expression is:

[0129]

[0130] where y i is the label value, p i is the model prediction value, and λ is the weight coefficient used to balance the two terms.

[0131] The value range of λ is set to 0.1 to 1.0 to balance the impact of cross entropy loss and Dice coefficient on the overall loss.

[0132] A training sample is set, containing the label values ​​y=[1,0,1,1,0] of 5 pixels and the corresponding predicted values ​​p=[0.9,0.2,0.8,0.7,0.1]. The combined loss function of the cross entropy loss and the Dice coefficient is calculated, and λ=0.5 is set.

[0133] Calculate the cross entropy loss:

[0134]

[0135] Calculate the Dice coefficient:

[0136]

[0137] DiceLoss=1-DiceCoefficient=1-0.948=0.052

[0138] Combined loss function:

[0139] L=Cross-Entropy+λ·DiceLoss

[0140] L=0.685+0.5·0.052

[0141] L=0.685+0.026

[0142] L=0.711

[0143] The embodiment further uses the Adam optimizer to adjust the model weights, and its update rule is:

[0144]

[0145] Among them, η t is the learning rate at time t, and are the bias-corrected first- and second-order moment estimates at time t, and ∈ is a small quantity to prevent division by zero.

[0146] η t The initial value of is set to 0.001;

[0147] β1 = 0.9 and β2 = 0.999 are the decay rates of the momentum term;

[0148] ∈=10 -8 is a small amount to prevent division by zero.

[0149] Set the current weight w in a certain training t is 0.5, and the gradient g t is 0.02. The initial first-order moment estimate m t and the second moment estimate v t are 0.0 respectively.

[0150] Compute first- and second-order moment estimates:

[0151] m t =β1·m t-1 +(1-β1)·g t

[0152] m t =0.9·0+0.1·0.02=0.002

[0153]

[0154] v t =0.999·0+0.001·(0.02) 2 =0.0000004

[0155] Bias correction:

[0156]

[0157] Weight update:

[0158]

[0159] w t+1 =0.5-0.001

[0160] w t+1 =0.499

[0161] Through the above calculations, we can see that in the t-th step of training, the weight is updated from 0.5 to 0.499.

[0162] The last example introduces a learning rate decay strategy to gradually reduce the learning rate to ensure the stability and accuracy of the model in the later stage of training. The specific learning rate decay strategy formula is:

[0163] η t+1 =η t ·e -αt

[0164] Among them, η t is the learning rate at time t, and α is the decay rate.

[0165] The initial learning rate η0 is set to 0.001; the value range of α is set to 0.001 to 0.01, which is used to control the decay rate of the learning rate.

[0166] The initial learning rate η0 = 0.001 and the decay rate α = 0.005 are set during the training process.

[0167] Initial learning rate:

[0168] η0=0.001

[0169] Learning rate at the first moment:

[0170] η1=η0·e -α·1

[0171] η1=0.001·e -0.005·1

[0172] η1=0.001·e -0.005

[0173] η1≈0.001·0.995

[0174] η1≈0.000995

[0175] Learning rate at the second moment:

[0176] η2=η1·e -α·2

[0177] η2=0.000995·e -0.005·2

[0178] η2=0.000995·e -0.01

[0179] η2≈0.000995·0.990

[0180] η2≈0.000985

[0181] From the above calculations, we can see that the learning rate is gradually decreasing, which helps to converge quickly in the early stages of training and adjust model parameters in the later stages of training to avoid overfitting.

[0182] Embodiment 3:

[0183] Combined with Figure 3 , the embodiment first adopts a model with a residual network structure and operates through the following feature extraction formula:

[0184]

[0185] Where x represents the input image data, W l and b l are the weights and biases of the lth layer, α l is the adjustment coefficient of each layer, L is the number of layers in the network, and * represent gradient and convolution operations respectively.

[0186] α l The value range of is set to 0.1 to 1.0, which is used to adjust the influence of each layer feature;

[0187] The number of network layers L is set to 5 to ensure sufficient depth to extract complex features;

[0188] Deviation b l The initial value range is set to [-0.01, 0.01];

[0189] Weight W l The initial value range is set to [-0.05, 0.05].

[0190] Set to process an ear canal and eardrum image of size 256x256 pixels. Specific calculation of the feature extraction process of the third layer:

[0191] Input image data: x is the preprocessed 256x256 pixel image.

[0192] Convolution operation and activation function:

[0193] W3*x+b3

[0194] The convolution kernel size is set to 3x3, the initial weight W3 is 0.02, and the bias b3 is 0.005. The pixel value of a certain area is x=[150,160,170,180,190,200,210,220,230].

[0195] Compute the weighted sum of the convolution operation:

[0196] W3*x=0.02·150+0.02·160+0.02·170+0.02·180+0.02·190+0.02·200+0.02·210+0.02·220+0.02·230+0.005

[0197] =3+3.2+3.4+3.6+3.8+4+4.2+4.4+4.6+0.005

[0198] =33.205

[0199] Apply the ReLU activation function:

[0200] ReLU(33.205)=max(0,33.205)=33.205

[0201] Set α3 = 0.8 and calculate the feature extraction value:

[0202]

[0203] Set the gradient calculation to 1:

[0204] ε(x)=0.8·33.205

[0205] =26.564

[0206] Calculate the total feature extraction value:

[0207] Similar calculations are performed on other regions of the third layer feature extraction to obtain the feature extraction values ​​of each region, which are finally summed and averaged. The final result is set as:

[0208]

[0209] Assume there are N regions, ε total (x)=27.34.

[0210] The embodiment further introduces a multi-level feature fusion technology to integrate feature maps at different levels through a feature fusion function:

[0211]

[0212] where y i is the feature map output by the i-th layer, β i ,γ i ,δ i It is a fusion adjustment parameter used to enhance the model's adaptability to different anatomical structure sizes and shapes.

[0213] β i The value range of is set to 0.5 to 2.0, which is used to control the weights of feature maps at different levels;

[0214] γ i The value range of is set to 0.1 to 2.0, which is used to adjust the nonlinear transformation of the feature map;

[0215] δ i The value range of is set to 0.5 to 1.5, which is used to control the enlargement or reduction of the fusion result.

[0216] Suppose that an ear canal and eardrum image with a size of 256x256 pixels is processed, and the three-layer feature map y1, y2, y3 is obtained through the feature extraction module:

[0217] The size of y1 is 64x64 pixels;

[0218] The size of y2 is 128x128 pixels;

[0219] The size of y3 is 256x256 pixels.

[0220] Set the fusion adjustment parameters as follows:

[0221] β1=1.0,γ1=1.2,δ1=0.8

[0222] β2=0.8,γ2=1.0,δ2=1.0

[0223] β3=1.2,γ3=0.9,δ3=1.1

[0224] Calculate the fusion value of the first layer feature map:

[0225]

[0226] Set the pixel value of a certain area of ​​y1 to y1 = [1.2, 1.5, 1.1, 0.9],

[0227]

[0228] Calculate the fusion value of the second layer feature map:

[0229]

[0230] Set the pixel value of a certain area of ​​y2 to y2 = [2.0, 1.8, 2.2, 2.1],

[0231]

[0232] Calculate the fusion value of the third layer feature map:

[0233]

[0234] Set the pixel value of a certain area of ​​y3 to y3 = [1.0, 1.3, 1.4, 1.2],

[0235]

[0236] =6.366

[0237] Calculate the total feature fusion value:

[0238]

[0239] Then the optimization function is used to improve the prediction ability of the model. The specific optimization function is:

[0240]

[0241] where z j is the component of the feature vector after dimensionality reduction, θ j is the optimization weight, k j is a nonlinear adjustment factor. Process an ear canal and tympanic membrane image of size 256x256 pixels, and obtain the feature vector z=[0.8,1.2,0.5] after dimension reduction after feature extraction and fusion. Set the optimization weights and nonlinear adjustment factors as follows: θ1=0.5, k1=1.0, θ2=0.8, k2=1.5, θ3=0.3, k3=0.5.

[0242] First, calculate the internal and external sums of the optimization function:

[0243]

[0244] Calculate each term:

[0245] 0.5 0.8 = 0.4

[0246] 0.8·(1.2 1.5 )≈0.8·1.39≈1.112

[0247] 0.3·(0.5 0.5 )≈0.3·0.707≈0.212

[0248] Add all the terms together:

[0249] 0.4+1.112+0.212≈1.724

[0250] Then evaluate the external function:

[0251]

[0252] Calculate the exponential part:

[0253] e -1.724 ≈0.178

[0254] The final optimized function value is:

[0255]

[0256] Through these detailed steps and specific calculations, the tympanic membrane puncture assisted navigation system based on image processing and machine learning significantly improves the accuracy and real-time performance of image analysis and path planning in actual surgery, thereby improving the safety and success rate of the surgery.

[0257] Embodiment 4:

[0258] The embodiment constructs a data collection and preprocessing module, applies a custom image transformation and enhancement algorithm to optimize image quality and ensure the accuracy of subsequent processing. The specific algorithm is expressed as follows:

[0259]

[0260] Where I(x,y) represents the input image frame, represents the gradient of the image at point (x, y), κ is the parameter for adjusting the local contrast sensitivity of the image, and Ω is the domain of the image.

[0261] In ear surgery, the patient's ear canal and tympanic membrane are captured in real time, and the image size is 256x256 pixels. In order to improve the quality and contrast of the image, an image transformation and enhancement algorithm is applied. The value range of κ is set to 0.01 to 0.1, and κ is taken here as 0.05.

[0262] Calculate the gradient of the image at point (x,y) The pixel value of the area is:

[0263]

[0264] Use the Sobel operator to calculate the gradient:

[0265]

[0266] The gradient at the center point 2,2 is calculated as:

[0267]

[0268] Apply image transformation and enhancement algorithms:

[0269]

[0270] At the center point 2,2:

[0271]

[0272] Apply the above transformation to the entire image:

[0273]

[0274] The integral result set on the entire image is:

[0275]

[0276] After processing the entire image in this way, an enhanced image is obtained, which can more clearly show the details of the ear canal and eardrum, thereby improving the accuracy of subsequent path planning.

[0277] The embodiment further uses a reinforcement learning model to dynamically generate and adjust the puncture path, and uses a policy update formula to optimize the model. Assume that at a certain time step t, the current policy π old The reward r t is 1.2, the discount factor γ is 0.95, and the learning rate α is 0.05.

[0278] Calculate the discount and total reward for future rewards:

[0279]

[0280] Setting rt In the next few time steps, they are 1.2, 1.1, and 1.3 respectively:

[0281]

[0282] Setting the gradient to the policy is 0.5,

[0283]

[0284] Update strategy:

[0285] π new =π old +0.0854

[0286] Through these steps, the reinforcement learning model can continuously learn and optimize during the surgery, improving the accuracy and flexibility of path planning.

[0287] Embodiment 5:

[0288] The embodiment uses a high-resolution endoscopic camera to capture images of the ear canal and tympanic membrane in real time, and applies real-time image processing techniques, including edge enhancement, contrast adjustment, dynamic noise filtering and illumination compensation, using a custom image processing algorithm:

[0289]

[0290] Where I(ξ,η) represents the intensity of the image at the coordinate (ξ,η), and σ is the Gaussian smoothing parameter used for edge enhancement and contrast adjustment.

[0291] First, a high-resolution endoscope camera is used to capture images of the ear canal and tympanic membrane in real time. The acquired images are 256x256 pixels. In order to enhance the edges and contrast of the images, a custom image processing algorithm is applied.

[0292] Set the pixel value of the captured image area to:

[0293]

[0294] The value range of the Gaussian smoothing parameter σ is set to 1 to 3, and σ=2 is taken here.

[0295] Compute edge enhancement and contrast adjustment:

[0296]

[0297] At the center point 2,2, the Gaussian smoothing and the second derivative are calculated:

[0298] Compute the second-order derivative:

[0299]

[0300] For center point 2,2:

[0301]

[0302] Compute Gaussian smoothing weights:

[0303]

[0304] For surrounding points 1,1 to 3,3:

[0305]

[0306] Compute the weighted sum:

[0307]

[0308] Apply the above algorithm to the entire image to get the enhanced image. Set the effect of the entire image after processing as follows:

[0309]

[0310] This processing method significantly enhances the edges of the image, making the tympanic membrane and surrounding structures clearer, and providing high-quality image data for subsequent path planning.

[0311] The next step of the embodiment is to implement a dynamic noise filtering and illumination compensation algorithm, and the specific algorithm is expressed as follows:

[0312]

[0313] Where I(t) is the image data at time t, μ(t) is the average brightness of the image at time t, and γ(t) is the dynamic compensation coefficient adjusted according to the change of ambient light.

[0314] The processed image size is set to 256x256 pixels, the time range T is 10, the value range of γ(t) is set to 0.8 to 1.2, which is used to dynamically adjust according to the changes in ambient light, and μ(t) is the average brightness of each frame of the image.

[0315] Assume that the image data I(t) captured at a certain time t is:

[0316]

[0317] And the average brightness μ(t) at this moment is:

[0318]

[0319] The motion compensation coefficient γ(t) is set to 1.1 at this moment.

[0320] Apply dynamic noise filtering and illumination compensation algorithms:

[0321]

[0322] Calculate the adjustment value for each pixel:

[0323]

[0324] Add the compensated values ​​back to the original image:

[0325]

[0326]

[0327] After the above processing, the contrast and brightness of the image are optimized and the noise is effectively filtered out.

[0328] Embodiment 6:

[0329] The embodiment uses a deep learning network and a reinforcement learning algorithm to dynamically generate and adjust the puncture path according to real-time updated image data, and applies a custom feature extraction formula:

[0330]

[0331] Among them I i is the i-th pixel in the image, represents the Laplace operator of the pixel, w i is the weighting factor and σ is the standard deviation of the adjusted response strength.

[0332] Parameter settings:

[0333] The value range of σ is set to 0.5 to 1.5, which is used to adjust the response strength; w i The value range is set to 0.1 to 1.0, which is used to control the weight of each pixel.

[0334] Set the pixel value of the captured image area to:

[0335]

[0336] Compute the Laplacian of this region:

[0337]

[0338] Set the weight factor w i is 0.5, and σ is 1.0.

[0339] Apply the feature extraction formula:

[0340]

[0341] Calculate the eigenvalues ​​for each pixel:

[0342] for

[0343]

[0344] for

[0345]

[0346] for

[0347]

[0348] Other pixels are similar, and most of the calculation results are close to 0, and the only eigenvalue with a value of 1 is:

[0349]

[0350] Extract key feature points from the processed image, use the reinforcement learning algorithm to train the path planning model, and generate the optimal puncture path. The reinforcement learning strategy update formula is:

[0351]

[0352] The value range of α is set to 0.01 to 0.1, which is used to control the learning rate; the value range of γ is set to 0.8 to 0.99, which is used to adjust the impact of future rewards; it is set that at a certain time step t, the current strategy π old The reward r t is 1.2, the discount factor γ is 0.95, and the learning rate α is 0.05.

[0353] Calculate the discount and total reward for future rewards:

[0354]

[0355] Setting r t In the next few time steps, they are 1.2, 1.1, and 1.3 respectively:

[0356]

[0357] Setting the gradient to the policy is 0.5,

[0358]

[0359] Update strategy:

[0360] π new =π old+0.0854

[0361] During the actual surgery, the system dynamically adjusts the puncture path based on the real-time captured images and the path generated by the model.

[0362] Embodiment 7:

[0363] Combined with Figure 4 In the embodiment, when performing ear surgery, the high-resolution endoscopic camera is used to capture images of the ear canal and eardrum in real time, and an image processing algorithm is applied for real-time tracking. The specific image processing algorithm is:

[0364]

[0365] I(x,y) is the image brightness function, is the Laplacian operator of the image at (x, y), which is used to detect edges and other high-frequency details. τ is the preset threshold, σ is the parameter for adjusting detail sensitivity, and Ω is the domain of the image.

[0366] The value range of τ is set to 0.5 to 1.5; the value range of σ is set to 0.5 to 1.5.

[0367] Processing a 256x256 pixel image of the ear canal and eardrum, the pixel values ​​of the area are:

[0368]

[0369] Compute the Laplacian of this region:

[0370]

[0371] The preset threshold τ is set to 1.0 and σ is set to 1.0.

[0372] Apply image processing algorithms:

[0373]

[0374] Calculate the detail detection value for each pixel:

[0375] for

[0376]

[0377] for

[0378]

[0379] for

[0380]

[0381] Most of the calculation results are close to 0, with only some details being more obvious:

[0382]

[0383] The embodiment further combines the dynamic deviation analysis technology, and the specific algorithm is:

[0384] Δ(s)=∫ s ρ(s,a)(κ(s,a,s′)-ν(s,a))ds′

[0385] where ρ(s,a) is the probability density function of state s and action a, κ(s,a,s′) is the expected reward when achieving state s′, ν(s,a) is the deviation cost function between the expected path and the actual path, and S represents the space of all states.

[0386] The value range of ρ(s,a) is set to 0.1 to 1.0, which is used to describe the probability density of different states and actions;

[0387] The value range of κ(s,a,s′) is set to 0.5 to 1.5, which is used to describe the expected reward when achieving state s′;

[0388] The value range of ν(s,a) is set to 0.1 to 1.0, which is used to describe the deviation cost between the expected path and the actual path.

[0389] Assume that in a certain state s, the probability density ρ(s,a) of the action a taken is 0.8, the expected reward κ(s,a,s′) is 1.2, the deviation cost ν(s,a) is 0.5, and calculate the dynamic deviation Δ(s).

[0390] Calculate the difference between expected reward and deviation cost:

[0391] κ(s,a,s′)-ν(s,a)=1.2-0.5=0.7

[0392] Compute the weighted sum of probability densities:

[0393] Δ(s)=∫ s ρ(s,a)·0.7ds′

[0394] Set the state space S to 1,

[0395] Δ(s)=0.8·0.7=0.56

[0396] The final embodiment uses a feedback control formula to dynamically adjust the action of the surgical tool, the formula is:

[0397]

[0398] Where λ is the learning rate, π(s,a) represents the strategy of taking action a in state s, γ is the discount factor, and Q(s,a) is the value function of action a in state s.

[0399] The value range of λ is set to 0.01 to 0.1 to control the learning speed; the value range of γ is set to 0.8 to 0.99 to adjust the impact of future rewards; π(s,a) is determined based on historical data and the training results of the machine learning model.

[0400] Assume that at a certain stage of surgery, the strategy π(s,a) and value function Q(s,a) under state s have been derived through a deep learning model. For example:

[0401] In state s, there are two actions a1 and a2.

[0402] π(s,a1)=0.7, π(s,a2)=0.3

[0403] Q(s,a1)=1.5,Q(s,a2)=1.8

[0404] Set γ = 0.95, λ = 0.05, and calculate the expected value difference of each action:

[0405] For a1:

[0406]

[0407] For a2:

[0408]

[0409] Calculate the total adjustment:

[0410] φ(s)=0.05(0.7·0.21+0.3·(-0.09))

[0411] φ(s)=0.05(0.147-0.027)

[0412] φ(s)=0.05·0.12=0.006

[0413] According to the changes in the adjustment value, the action of the surgical tool can be adjusted in real time.

[0414] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. Tympanic membrane puncture auxiliary navigation system based on image processing and machine learning, characterized by include: The image segmentation module uses a convolutional neural network to segment the ear canal and tympanic membrane images to separate the tympanic membrane and its surrounding anatomical structures; The convolutional neural network architecture adopts the U-Net network architecture. The encoder part extracts the high-dimensional features of the image layer by layer, and the decoder part gradually restores the spatial information of the image through jump connections. Then, the cross entropy loss function and the Dice coefficient are used as loss functions to balance the accuracy of the segmentation results and the regional overlap. The Adam optimizer is used to optimize the model parameters and the learning rate decay strategy is used to prevent overfitting. The feature extraction module uses a deep learning model to extract key feature points from the segmented image, including the morphological features, position and surrounding structures of the tympanic membrane. The path planning module collects surgical videos and puncture path data and uses a reinforcement learning algorithm to train the path planning model, so that the path planning module generates the optimal puncture path in the image; the real-time path adjustment module uses real-time image processing technology to update the images of the ear canal and tympanic membrane, dynamically adjusts the puncture path through the path planning model, and controls the accuracy of the path during the operation; the navigation interface module develops a surgical navigation interface, displays high-definition images, puncture paths and related anatomical structures in real time, and integrates 3D views to provide multi-angle observation of surgical scenes; The convolutional neural network architecture design process is as follows: S1, using the U-Net network architecture to extract and segment the image features. The encoder part extracts the high-dimensional features of the image layer by layer, using the convolution operation formula: C(x,k)=∫ Ω σ(W(k) x(tk)+b(k))dk Where x represents the input image data, W(k) and b(k) are position-dependent learning parameters, σ is the activation function, t is the variable in the convolution operation, and Ω is the entire input domain; S2. Custom feature depth adjustment function: F(d,z)=α·d 2 ·e -λ·z Where d represents the depth of the network, z is the complexity evaluation of the input features, and α and λ are adjustment coefficients to optimize the feature extraction depth of each layer and adapt to different input complexities; S3, the decoder part gradually restores the spatial information of the image through upsampling and convolutional layers, and uses jump connections to enhance the retention of structural details. The corresponding spatial information recovery formula is: R(y)=∫ s β·y (γs) ds Where y represents the feature map passed from the encoder to the decoder, β and γ are adjustment coefficients that control the fineness of recovery and the degree of information retention, and S is the spatial range of feature recovery; The accuracy of the balanced segmentation results and the area overlap are achieved by combining the cross entropy loss and the Dice coefficient, which is specifically expressed as the loss function: where y i is the label value, p i is the model prediction, and λ is the weight coefficient used to balance the two terms; The Adam optimizer is used to adjust the model weights, where the model weight update rule is: η t is the learning rate at time t, and are the bias-corrected first-order and second-order moment estimates at time t, ∈ is a small amount to prevent division by zero; And implement the learning rate decay strategy, specifically: or t+1 =the t ·e -αt η t is the learning rate at time t, and α is the decay rate.

2. The tympanic membrane puncture auxiliary navigation system based on image processing and machine learning according to claim 1, characterized in that The feature extraction module includes: firstly adopting a model with a residual network structure and operating through a feature extraction formula: Where x represents the input image data, W l and b l are the weights and biases of the lth layer, α l is the adjustment coefficient of each layer, L is the number of layers in the network, and * represent gradient and convolution operations respectively; Then implement the multi-level feature fusion technology through the feature fusion function: where y i is the feature map output by the i-th layer, β i ,γ i , and δ i It is to integrate adjustment parameters to enhance the model’s adaptability to different anatomical structure sizes and shapes; The feature vector is post-processed using the optimization function: where z j is the component of the feature vector after dimensionality reduction, θ j is the optimization weight, k j is a nonlinear adjustment factor.

3. The tympanic membrane puncture auxiliary navigation system based on image processing and machine learning according to claim 1, characterized in that The path planning module construction method includes: using a data collection and preprocessing module, applying a custom image transformation and enhancement algorithm, expressed as: Where I(x,y) represents the input image frame, represents the gradient of the image at point (x, y), κ is the parameter for adjusting the local contrast sensitivity of the image, and Ω is the domain of the image; The design and training module of the reinforcement learning model uses a complex strategy update formula: where π old and π new They represent the strategies before and after the update, α is the learning rate, γ is the discount factor that affects the current value of future rewards, and r t is the reward obtained at time t, represents the gradient of the policy.

4. The tympanic membrane puncture auxiliary navigation system based on image processing and machine learning according to claim 1, characterized in that The path planning module construction process is as follows: S1. Use a high-resolution endoscopic camera to capture images of the ear canal and tympanic membrane in real time and apply real-time image processing techniques, including edge enhancement, contrast adjustment, dynamic noise filtering, and illumination compensation; S2, using deep learning networks and reinforcement learning algorithms to dynamically generate and adjust the puncture path based on real-time updated image data, and represent new image information and surgical dynamics based on the original path planning; S3, integrated real-time feedback system, directly controls surgical tools through real-time tracking and deviation analysis of puncture needle position.

5. The tympanic membrane puncture auxiliary navigation system based on image processing and machine learning according to claim 4, characterized in that The real-time capture of the images of the ear canal and the eardrum uses an image processing algorithm: Where I(ξ,η) represents the intensity of the image at the coordinate (ξ,η), and σ is the Gaussian smoothing parameter used for edge enhancement and contrast adjustment; And implement dynamic noise filtering and illumination compensation algorithms: Where I(t) is the image data at time t, μ(t) is the average brightness of the image at time t, and γ(t) is the dynamic compensation coefficient adjusted according to the change of ambient light.

6. The tympanic membrane puncture auxiliary navigation system based on image processing and machine learning according to claim 4, characterized in that The implementation process of the deep learning network and reinforcement learning algorithm is as follows: First, the deep learning network is used to process and analyze the real-time captured ear canal and tympanic membrane images, and a custom feature extraction formula is applied: Among them I i is the i-th pixel in the image, represents the Laplace operator of the pixel, w i is the weight factor, σ is the standard deviation of the adjusted response intensity; Then, the reinforcement learning algorithm is combined to dynamically generate and adjust the puncture path, using the dynamic path optimization formula: where π old and π new represents the old strategy and the new strategy respectively, s and s′ are the current and next states, a is the action taken, P(s′|s,a) is the transition probability from state s to state s′ through action a, R(s,a,s′) is the reward function, γ is the discount factor for future rewards, Q(s,a) is the action value function, and η is the learning rate.

7. The tympanic membrane puncture auxiliary navigation system based on image processing and machine learning according to claim 4, characterized in that The integrated real-time feedback system construction process is as follows: S1. Use real-time tracking technology, which is specifically achieved by capturing images with a high-resolution endoscope camera and applying image processing algorithms: Where I(x,y) is the image brightness function, is the Laplacian operator of the image at (x, y), which is used to detect edges and other high-frequency details, τ is the preset threshold, σ is the parameter for adjusting detail sensitivity, and Ω is the domain of the image; S2. Combined with dynamic deviation analysis technology, the specific algorithm is: Δ(s)=∫ s ρ(s,a)(κ(s,a,s′)-ν(s,a))ds′ Where ρ(s,a) is the probability density function of state s and action a, κ(s,a,s′) is the expected reward when achieving state s′, ν(s,a) is the deviation cost function between the expected path and the actual path, and S represents all state spaces; S3, and integrated control system, through the feedback control formula: Adjust the actions of the surgical tool, where λ is the learning rate, π(s,a) represents the policy of taking action a in state s, γ is the discount factor, and Q(s,a) is the value function of action a in state s.

Images