Endoscope image processing method and system

Abstract

The invention provides an endoscope image processing method and system, and the method comprises the steps: obtaining the electroencephalogram data and eye movement data of an operator, and aligning the time stamps of the electroencephalogram data, the eye movement data and an endoscope video stream; according to the electroencephalogram data and the eye movement data, electroencephalogram features and eye movement features are extracted; the features are input into a pre-trained radial basis function neural network, adjustment parameters are output, a reinforcement learning optimizer is adopted, it is ensured that output of the radial basis function neural network is optimal, the reinforcement learning optimizer is achieved through a DDPG algorithm, the reward function is composed of a fatigue relief reward, an attention optimization reward and an operation smooth reward, and the reward function is used for rewarding the fatigue relief reward, the attention optimization reward and the operation smooth reward. Determining a fatigue relief reward according to the electroencephalogram features, and determining an attention optimization reward according to the eye movement features; according to the method, the endoscope image in the endoscope video stream is adjusted according to the adjustment parameter, and specifically, the physiological reaction of the operator is converted into a quantifiable optimization instruction, so that personalized adjustment of the endoscope image watched by the operator is realized.

Description

Technical Field

[0001] This invention belongs to the field of endoscopic image processing technology, and specifically relates to an endoscopic image processing method and system. Background Technology

[0002] An endoscope is a diagnostic and treatment instrument that is inserted through a natural opening or incision in the human body to view the inside of the body. It can be divided into rigid endoscopes and flexible endoscopes, and is mainly used for diagnosis and treatment in clinical practice.

[0003] The current technical challenges in endoscopic image processing include potential issues such as image blurring, low contrast, color distortion, or noise. It also faces challenges such as insufficient noise processing, difficulty in highlight removal, large amounts of data to be processed in real time with high hardware performance requirements, and the need for algorithms to adapt to different devices and changing clinical environments.

[0004] To address this, existing technologies employ advanced denoising algorithms such as wavelet transform to suppress image noise, utilize techniques like histogram equalization to improve image contrast, employ super-resolution techniques to enhance image resolution, optimize image processing algorithms, establish real-time feedback mechanisms, and use high-performance processors and storage devices to meet real-time processing requirements. It should be noted that although image quality has been significantly improved, the image processing process does not take into account the surgeon's visual fatigue level. In other words, the current improvement in image quality cannot alleviate visual fatigue for different surgeons and cannot achieve personalized adaptive optimization. Summary of the Invention

[0005] Based on this, the present invention provides an endoscopic image processing method and system, which aims to solve the problem that in the prior art, the improvement of image quality cannot alleviate the visual fatigue of different surgeons.

[0006] A first aspect of the present invention provides an endoscopic image processing method, the method comprising: Acquire the surgeon's EEG and eye movement data, and align the timestamps of the EEG data, the eye movement data, and the endoscopic video stream. Based on the electroencephalogram (EEG) data and the eye movement data, neural features are extracted, including EEG features extracted from the EEG data and eye movement features extracted from the eye movement data. The neural features are input into a pre-trained radial basis function neural network, and the output is adjusted. A reinforcement learning optimizer is used to ensure that the output of the radial basis function neural network is optimal. The reinforcement learning optimizer is implemented by the DDPG algorithm. The reward function of the reinforcement learning optimizer consists of fatigue relief reward, attention optimization reward and operation fluency reward. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the eye movement features. The endoscopic images in the endoscopic video stream are adjusted according to the adjustment parameters.

[0007] Furthermore, the EEG data is extracted based on occipital and parietal electrode groups, and the EEG features include alpha band relative power, theta band energy entropy, P300 wave average amplitude, alpha band asymmetry index, beta band event-related desynchronization value, and transelectrode coherence; the eye movement data is extracted based on fixation and saccade patterns, and the eye movement features include average fixation duration, saccade amplitude standard deviation, red zone fixation percentage, pupil diameter variation coefficient, attention center shift, and fixation point density peak.

[0008] Furthermore, the adjustment parameters include red zone suppression intensity, tissue edge enhancement coefficient, dynamic range expansion amplitude, local contrast gain, color compensation value, high-frequency noise filtering threshold, central region magnification coefficient, and motion blur correction intensity.

[0009] Furthermore, the radial basis function neural network includes an input layer, a hidden layer, and an output layer. The input layer has 12 neurons, and the input layer and the hidden layer are fully connected. The hidden layer has 36 neurons and uses Gaussian radial basis functions as activation functions. The output layer has 8 neurons, and the output result is calculated by linear weighted summation. The output range is constrained to [-0.05, 0.05] by linear mapping.

[0010] Furthermore, the state variables in the state space of the reinforcement learning optimizer include the neural features, image features, and operational features. The image features include the proportion of red area pixels, average image brightness, high-frequency edge energy, dynamic range coverage, and color deviation index. The operational features include instrument movement speed, surgical stage labels, and historical adjustment feedback. The action variables in the action space of the reinforcement learning optimizer include the red zone suppression intensity adjustment, tissue edge enhancement coefficient adjustment, dynamic range expansion amplitude adjustment, local contrast gain adjustment, color compensation value adjustment, high-frequency noise filtering threshold adjustment, central region magnification coefficient adjustment, and motion blur correction intensity adjustment.

[0011] Furthermore, the network structure of the DDPG algorithm includes an Actor network and a Critic network, with an experience replay pool capacity of 5×10^4 experiences and 32 batches of random sampling for training. The Actor network is used to input state variables and output action variables, while the Critic network is used to input state variables and action variables and output Q-values, employing the MSE loss function.

[0012] Furthermore, the expression for the reward function is: R_immediate = 0.3×R_fatigue + 0.4×R_attention + 0.3×R_operation; R_fatigue = 1 - FI; FI = 0.6 × (α-wave power / (α-wave power + β-wave power)) + 0.4 × θ-band energy entropy; R_operation = 1 - Equipment movement jitter rate; R_immediate is the reward function; R_fatigue is the fatigue relief reward; FI is the fatigue index, ranging from [0,1]; α-wave power and β-wave power are extracted based on the occipital and parietal electrode groups. R_attention is the attention optimization reward, which is related to eye-tracking data. The attention optimization reward is the sum of a first reward value, a second reward value, and a third reward value. The first reward value is determined when the fixation point falls on a key area marked before surgery. The second reward value is determined when the attention entropy value is greater than a threshold. The third reward value is determined when the average fixation duration is within a preset range. The eye-tracking data includes fixation point coordinates, duration, saccade amplitude, and velocity. A Gaussian kernel function is used to transform the fixation point distribution into an attention heatmap. The weights are positively correlated with fixation duration. The attention entropy value is obtained by calculating the pixel value distribution of the attention heatmap. R_operation is the reward for smooth operation, and the device motion jitter rate is the device trajectory fluctuation rate calculated based on optical flow.

[0013] A second aspect of the present invention provides an endoscopic image processing system for implementing the endoscopic image processing method described in the first aspect, the system comprising: The acquisition module is used to acquire the surgeon's EEG data and eye movement data, and to align the timestamps of the EEG data, the eye movement data, and the endoscopic video stream. An extraction module is used to extract neural features based on the electroencephalogram (EEG) data and the eye movement data, wherein the neural features include EEG features extracted from the EEG data and eye movement features extracted from the eye movement data; The input module is used to input the neural features into a pre-trained radial basis function neural network and output adjustment parameters. A reinforcement learning optimizer is used to ensure that the output of the radial basis function neural network is optimal. The reinforcement learning optimizer is implemented by the DDPG algorithm. The reward function of the reinforcement learning optimizer consists of fatigue relief reward, attention optimization reward and operation fluency reward. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the eye movement features. An adjustment module is used to adjust the endoscopic image in the endoscopic video stream according to the adjustment parameters.

[0014] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the endoscopic image processing method provided in the first aspect.

[0015] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the endoscopic image processing method provided in the first aspect.

[0016] This invention provides an endoscopic image processing method and system that acquires the surgeon's electroencephalogram (EEG) and eye-tracking (EOL) data, aligning the timestamps of the EEG, EOL, and endoscopic video stream. Based on the EEG and EOL data, neural features are extracted, including EEG features extracted from the EEG data and EOL features extracted from the EOL data. These neural features are input into a pre-trained radial basis function (RBF) neural network, which outputs adjustment parameters. A reinforcement learning optimizer is used to ensure optimal output of the RBF neural network. The RBF optimizer is implemented using the DDPG algorithm, and its reward function consists of fatigue relief rewards, attention optimization rewards, and operational fluency rewards. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the EOL features. The endoscopic images in the endoscopic video stream are adjusted according to the adjustment parameters. Specifically, by converting the surgeon's physiological responses into quantifiable optimization instructions, personalized adjustment of the endoscopic images viewed by the surgeon is achieved. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the implementation of an endoscopic image processing method according to Embodiment 1 of the present invention. Figure 2 This is a structural block diagram of an endoscope image processing system provided in Embodiment 2 of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0018] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0019] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0021] Example 1 According to an embodiment of the present invention, an embodiment of an endoscopic image processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0022] This first embodiment provides an endoscopic image processing method that can be used in electronic devices, such as computers. Please refer to... Figure 1 , Figure 1 The flowchart of an endoscopic image processing method according to Embodiment 1 of the present invention is shown, specifically including steps S01 to S04.

[0023] Step S01: Obtain the surgeon's EEG data and eye movement data, and align the timestamps of the EEG data, the eye movement data, and the endoscopic video stream.

[0024] In this embodiment of the invention, a dry electrode array is used, distributed in the O1 / O2 / Oz region of the occipital lobe and the P3 / P4 region of the parietal lobe, to acquire electroencephalogram (EEG) data. Eye movement data is acquired by integrating an infrared CMOS camera and a near-infrared light source using the corneal reflection-pupil center vector method.

[0025] The study involved power spectrum analysis of EEG data, using the Welch method (256 points in the window, 128 points of overlap) to calculate the power of alpha waves (8-13Hz) and beta waves (14-30Hz), and extracting the energy entropy value of the theta band (4-7Hz). Based on the alpha wave power, beta wave power, and the energy entropy of the theta band, the fatigue relief reward in the subsequent reward function was evaluated. Eye movement data included the extraction of fixation point coordinates, duration (>100ms was considered a valid fixation), saccade amplitude, and velocity. Subsequently, a Gaussian kernel function (σ=1.5° field of view) was used to transform the fixation point distribution into an attention heatmap, with the weights positively correlated with fixation duration. This attention heatmap was used to evaluate the attention optimization reward in the subsequent reward function.

[0026] Step S02: Extract neural features based on the EEG data and the eye movement data. The neural features include EEG features extracted from the EEG data and eye movement features extracted from the eye movement data.

[0027] Specifically, the EEG data is extracted based on the occipital and parietal electrode groups, and the EEG features include alpha band relative power, theta band energy entropy, P300 wave mean amplitude, alpha band asymmetry index, beta band event-related desynchronization value, and transelectrode coherence; the eye movement data is extracted based on fixation and saccade patterns, and the eye movement features include mean fixation duration, saccade amplitude standard deviation, red zone fixation percentage, pupil diameter variation coefficient, attention center shift, and fixation point density peak.

[0028] It should be noted that the relative power of the alpha band refers to the proportion of power in the 8-13Hz frequency band to the total power (1-30Hz), reflecting the activity of the visual cortex; the energy entropy of the theta band refers to the sample entropy in the 4-7Hz frequency band, reflecting the irregularity of the EEG signal (the entropy value increases when fatigued); the average amplitude of the P300 wave refers to the average amplitude of event-related potentials in the 300-500ms window during image parameter fine-tuning (oddball stimulation), with a value range of [-10, 10] (positive values ​​indicate a preference for this adjustment); the alpha band asymmetry index refers to (alpha power of O1 electrode - alpha power of O2 electrode) / (O1+O2) reflects the difference in visual attention between the left and right hemispheres, with a value range of [-1, 1] (positive values ​​indicate that the left hemisphere is more active); the β-band event-related desynchronization value refers to the percentage decrease in power when fixating on key structures in the 14-30Hz frequency band, expressed as (baseline power - task power) / baseline power, with a value range of [-0.5, 0.5] (negative values ​​indicate suppression); transelectrode coherence refers to the coherence coefficient between the occipital (Oz) and apical (POz) electrodes in the 8-13Hz frequency band, reflecting visual network synergy, with a value range of [0, 1] (higher values ​​indicate better network synergy).

[0029] Mean fixation duration refers to the average duration of continuous fixation (speed <30° / s); saccade amplitude standard deviation refers to the angular standard deviation of continuous saccades (speed >30° / s), reflecting the stability of visual field search; red zone fixation percentage refers to the percentage of time the fixation point falls within the red zone (within the HSV red threshold), which can be understood as a hemorrhage / congestion area; pupil diameter coefficient of variation refers to the ratio of the standard deviation to the mean of the pupil diameter, reflecting the stability of light adaptation; attention center offset refers to the Euclidean distance between the current fixation point and the image center; fixation point density peak refers to the maximum pixel value of the attention heatmap (Gaussian weighted), reflecting the concentration of the attention area.

[0030] Step S03: Input the neural features into a pre-trained radial basis function neural network and output adjustment parameters.

[0031] Specifically, the adjustment parameters include red zone suppression intensity, tissue edge enhancement coefficient, dynamic range expansion amplitude, local contrast gain, color compensation value, high-frequency noise filtering threshold, central region magnification coefficient, and motion blur correction intensity. Understandably, adjusting the red zone suppression intensity targets highly saturated red areas in the image (such as hemorrhage or congested tissue), reducing the saturation and brightness of the red channel to suppress visual interference from excessively red areas while preserving tissue details; adjusting the tissue edge enhancement coefficient enhances areas with dramatic grayscale changes in the image (such as organ boundaries, fascia layers, and blood vessel walls), improving the clarity of tissue contours and helping surgeons identify anatomical structure boundaries; adjusting the dynamic range expansion amplitude expands the brightness range of the image, simultaneously improving detail preservation in highlight areas (such as reflective points) and brightness in shadow areas (such as tissue folds), resolving the coexistence of "overexposure" and "underexposure" in the endoscopic field of view; adjusting the local contrast gain adjusts the contrast of local areas in the image (such as small blood vessels and fine tissues), enhancing the distinguishability of local texture features and avoiding the loss of detail in some areas due to global contrast adjustment; color compensation value... The adjustment of the hue parameter is used to compensate for the overall color shift after red zone suppression and other processing, making the tissue color closer to the real state (such as the pinkness of mucous membranes and the yellowish-white of fat), and avoiding color deviation from affecting the identification of tissue type. The adjustment of the high-frequency noise filtering threshold is used to filter high-frequency noise in the image (such as sensor interference and graininess caused by light fluctuations). The higher the threshold, the greater the noise reduction intensity, balancing noise suppression and detail preservation (avoiding excessive blurring). The adjustment of the central region magnification coefficient is used to digitally magnify the central region of the surgeon's gaze (based on eye movement thermograms), increase the pixel density of the local area, and facilitate the observation of fine structures (such as small blood vessel branches and lesion details). The adjustment of the motion blur correction intensity is used to address the dynamic blur caused by instrument movement or body shaking. The motion trajectory is estimated by the algorithm and compensated in reverse to enhance the clarity of the blurred area, which is especially suitable for rapid operation scenarios (such as dissection and suturing).

[0032] The radial basis function neural network comprises an input layer, a hidden layer, and an output layer. The input layer has 12 neurons, corresponding to 6 EEG features and 6 eye-tracking features. The input layer and the hidden layer are fully connected (no weights, only feature values ​​are passed). The hidden layer has 36 neurons and uses the Gaussian radial basis function (RBF) as the activation function, which can be expressed as φ_j(x)=exp(-||x-c_j|| 2 / (2σ_j 2In the first step, x is a 12-dimensional input vector, c_j is the center vector of the j-th RBF neuron, and σ_j is the width parameter of the j-th RBF neuron. In the initialization step of the center vector, the k-means clustering algorithm is used to extract 36 cluster centers from the training data. In the setting step of the width parameter, σ_j=d_max / √(2N), where d_max is the distance between the j-th center and the nearest neighbor center, and N is the number of neurons in the hidden layer. The number of neurons in the output layer is 8, corresponding to 8 adjustment parameters. The output result is calculated by linear weighted summation and can be expressed as y_i=Σ(w_ij×φ_j(x))+b_i, where y_i is the i-th adjustment parameter, w_ij is the weight from the j-th neuron in the hidden layer to the i-th neuron in the output layer, and b_i is the output layer bias term. The output range is constrained to [-0.05, 0.05] by linear mapping, that is, ±5% adjustment.

[0033] In this embodiment of the invention, a reinforcement learning optimizer is used to ensure that the output of the radial basis function neural network is optimal. The reinforcement learning optimizer is implemented by the DDPG algorithm. The reward function of the reinforcement learning optimizer consists of fatigue relief reward, attention optimization reward, and operation fluency reward. The fatigue relief reward is determined based on the EEG characteristics, and the attention optimization reward is determined based on the eye movement characteristics.

[0034] It should be noted that the state variables in the state space of the reinforcement learning optimizer include the neural features, image features, and operational features. The image features include the percentage of red pixels (0-1.0), average image brightness (0-255, normalized to 0-1), high-frequency edge energy (sum of Sobel operator responses, normalized), dynamic range coverage ((max brightness - min brightness) / 255), and color deviation index (HSV distance between the current image and the standard tissue color library). The operational features include instrument movement speed (calculated based on optical flow, unit ° / s, normalized), surgical stage label (one-hot encoded: dissection / hemostasis / suture / exploration), and historical adjustment feedback (average reward value of the last 3 parameter adjustments). The action variables in the action space of the reinforcement learning optimizer include the adjustment amount of red zone suppression intensity, tissue edge enhancement coefficient, dynamic range expansion amplitude, local contrast gain, color compensation value, high-frequency noise filtering threshold, central region magnification coefficient, and motion blur correction intensity. In addition, an action constraint mechanism is set. For example, the hard constraint is that the adjustment of each parameter must not exceed the safe range, the smooth constraint is that the change in action between two consecutive frames is ≤0.03, and the priority constraint is that the weight of the red zone suppression intensity adjustment is increased by 30% in the hemostasis scenario and the weight of the tissue edge enhancement coefficient adjustment is increased by 20% in the stripping scenario.

[0035] In a reinforcement learning optimizer, the reward function determines the optimizer's learning direction and final result. The expression for the reward function is: R_immediate=0.3×R_fatigue+0.4×R_attention+0.3×R_operation; R_fatigue = 1 - FI; FI = 0.6 × (α-wave power / (α-wave power + β-wave power)) + 0.4 × θ-band energy entropy; R_operation = 1 - Equipment movement jitter rate; R_immediate is the reward function; R_fatigue is the fatigue relief reward; FI is the fatigue index, ranging from [0,1]; α-wave power and β-wave power are extracted based on the occipital and parietal electrode groups. R_attention is the attention optimization reward, which is related to eye-tracking data. The attention optimization reward is the sum of a first reward value, a second reward value, and a third reward value. The first reward value is determined when the fixation point falls on a pre-marked key area; the second reward value is determined when the attention entropy value is greater than a threshold; and the third reward value is determined when the average fixation duration is within a preset range. The eye-tracking data includes fixation point coordinates, duration, saccade amplitude, and velocity. A Gaussian kernel function is used to transform the fixation point distribution into an attention heatmap, with weights positively correlated with fixation duration. The attention entropy value is calculated from the pixel value distribution of the attention heatmap. For example, if the fixation point falls on a key area (pre-marked) → +0.5, the attention entropy value (distribution uniformity) > 0.6 → +0.3, and the average fixation duration is in the 300-600ms range → +0.2, it can be understood that when all the above conditions are met, R_attention = 0.5 + 0.3 + 0.2 = 1. R_operation is the reward for smooth operation, and the device motion jitter rate is the device trajectory fluctuation rate calculated based on optical flow.

[0036] In other embodiments of the present invention, the reward function may also include delayed rewards and penalty terms. Specifically, the delayed rewards include key structure recognition rewards and parameter stability rewards. In the key structure recognition reward, if the surgeon successfully locates the target structure within 5 seconds (determined by eye movement trajectory) → +2.0; in the parameter stability reward, if the total parameter adjustment amplitude within 5 seconds is <0.3 → +0.5. The penalty terms include tissue color deviation caused by excessive suppression of the red zone, for example, when the color deviation index is >0.2 → -0.8, and also include operational instability, for example, when the instrument movement jitter rate is >0.3 → -1.0.

[0037] Furthermore, the network structure of the DDPG algorithm includes an Actor network and a Critic network, with an experience replay pool capacity of 5 × 10^4 experiences, and employs quadruples. This indicates that, where s represents the current state. 'r' represents the action taken in state 's', and 'r' represents the action performed. The reward received later Indicates the execution of an action The system then transitions to the next state, and trains in 32 batches using random sampling. The Actor network is used to input state variables and output action variables, while the Critic network is used to input both state variables and action variables and output Q-values. The MSE loss function is employed.

[0038] More specifically, in the Actor network, the input layer is a 20-dimensional state vector, the hidden layers consist of three fully connected layers (128→64→32 nodes) with LeakyReLU activation, and the output layer is an 8-dimensional action vector with Tanh activation. L2 regularization is applied to each layer. The Actor loss is -E[Q(s,Actor(s))] (maximizing the Q-value).

[0039] In the Critic network, the state branches transition from 20 dimensions to 64 dimensions (ReLU activation); the action branches transition from 8 dimensions to 64 dimensions (ReLU activation); the fusion layer transitions from 64+64 to 128 dimensions (ReLU activation); and the output layer is a 1-dimensional Q-value (without an activation function). The Critic loss is... γ=0.99 (discount factor). For the target network of the Critic network, The target network for the Actor network.

[0040] The optimizer used is the Adam optimizer, with learning rates α_actor=1e-4, α_critic=1e-3, β1=0.9, and β2=0.999. α_actor and α_critic are the learning rates of the Actor network and the Critic network, respectively, and β1 and β2 are the exponential decay rates of the first and second moments estimated in the Adam algorithm, respectively.

[0041] Step S04: Adjust the endoscopic image in the endoscopic video stream according to the adjustment parameters.

[0042] Understandably, an endoscopic video stream consists of several endoscopic images, and each endoscopic image is processed according to the adjustment parameters.

[0043] In summary, the endoscopic image processing method in the above embodiments of the present invention acquires the surgeon's EEG and eye-tracking data and aligns the timestamps of the EEG, eye-tracking data, and endoscopic video stream. Based on the EEG and eye-tracking data, neural features are extracted, including EEG features extracted from the EEG data and eye-tracking features extracted from the eye-tracking data. These neural features are input into a pre-trained radial basis function neural network, which outputs adjustment parameters. A reinforcement learning optimizer is used to ensure optimal output of the radial basis function neural network. The reinforcement learning optimizer is implemented using the DDPG algorithm, and its reward function consists of fatigue relief rewards, attention optimization rewards, and operational fluency rewards. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the eye-tracking features. Based on the adjustment parameters, the endoscopic images in the endoscopic video stream are adjusted. Specifically, by converting the surgeon's physiological responses into quantifiable optimization instructions, personalized adjustment of the endoscopic images viewed by the surgeon is achieved.

[0044] Example 2 Please see Figure 2 , Figure 2 This is a structural block diagram of an endoscopic image processing system according to Embodiment 2 of the present invention. This endoscopic image processing system 200 is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0045] Specifically, the endoscopic image processing system 200 includes: an acquisition module 21, an extraction module 22, an input module 23, and an adjustment module 24, wherein: The acquisition module 21 is used to acquire the surgeon's electroencephalogram (EEG) data and eye movement data, and to align the timestamps of the EEG data, the eye movement data, and the endoscopic video stream. Extraction module 22 is used to extract neural features based on the EEG data and the eye movement data. The neural features include EEG features extracted from the EEG data and eye movement features extracted from the eye movement data. The EEG data is extracted based on occipital and parietal electrode groups. The EEG features include alpha band relative power, theta band energy entropy, P300 wave average amplitude, alpha band asymmetry index, beta band event-related desynchronization value, and cross-electrode coherence. The eye movement data is extracted based on fixation and saccade patterns. The eye movement features include average fixation duration, saccade amplitude standard deviation, red zone fixation percentage, pupil diameter variation coefficient, attention center shift, and fixation point density peak. Input module 23 is used to input the neural features into a pre-trained radial basis function neural network and output adjustment parameters. A reinforcement learning optimizer is used to ensure optimal output of the radial basis function neural network. The reinforcement learning optimizer is implemented using the DDPG algorithm. The reward function of the reinforcement learning optimizer consists of fatigue relief reward, attention optimization reward, and operational fluency reward. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the eye movement features. The adjustment parameters include red zone inhibition intensity, tissue edge enhancement coefficient, dynamic range expansion amplitude, local contrast gain, color compensation value, high-frequency noise filtering threshold, central region magnification coefficient, and motion blur correction intensity. Additionally, the radial basis function... The functional neural network includes an input layer, a hidden layer, and an output layer. The input layer has 12 neurons, and the input layer and the hidden layer are fully connected. The hidden layer has 36 neurons, using a Gaussian radial basis function as the activation function. The output layer has 8 neurons, and the output result is calculated by linear weighted summation. The output range is constrained to [-0.05, 0.05] by a linear mapping. The state variables in the state space of the reinforcement learning optimizer include neural features, image features, and operational features. The image features include the proportion of red area pixels, average image brightness, high-frequency edge energy, dynamic range coverage, and color deviation index. The operational features include instrument movement speed, surgical stage labels, and historical adjustment feedback. The action variables in the action space of the reinforcement learning optimizer include the red zone suppression intensity adjustment, tissue edge enhancement coefficient adjustment, dynamic range expansion amplitude adjustment, local contrast gain adjustment, color compensation value adjustment, high-frequency noise filtering threshold adjustment, central region magnification coefficient adjustment, and motion blur correction intensity adjustment. The network structure of the DDPG algorithm includes an Actor network and a Critic network. The experience replay pool has a capacity of 5×10^4 experiences, and 32 batches of training are randomly sampled. The Actor network is used to input state variables and output action variables, and the Critic network is used to input state variables and action variables and output Q-values. The MSE loss function is used. The expression for the reward function is: R_immediate = 0.3×R_fatigue + 0.4×R_attention + 0.3×R_operation; R_fatigue = 1 - FI; FI = 0.6 × (α-wave power / (α-wave power + β-wave power)) + 0.4 × θ-band energy entropy; R_operation = 1 - Equipment movement jitter rate; R_immediate is the reward function; R_fatigue is the fatigue relief reward; FI is the fatigue index, ranging from [0,1]; α-wave power and β-wave power are extracted based on the occipital and parietal electrode groups. R_attention is the attention optimization reward, which is related to eye-tracking data. The attention optimization reward is the sum of a first reward value, a second reward value, and a third reward value. The first reward value is determined when the fixation point falls on a key area marked before surgery. The second reward value is determined when the attention entropy value is greater than a threshold. The third reward value is determined when the average fixation duration is within a preset range. The eye-tracking data includes fixation point coordinates, duration, saccade amplitude, and velocity. A Gaussian kernel function is used to transform the fixation point distribution into an attention heatmap. The weights are positively correlated with fixation duration. The attention entropy value is obtained by calculating the pixel value distribution of the attention heatmap. R_operation is the reward for smooth operation, and the device motion jitter rate is the device trajectory fluctuation rate calculated based on optical flow.

[0046] The adjustment module 24 is used to adjust the endoscopic image in the endoscopic video stream according to the adjustment parameters.

[0047] Example 3 In another aspect, the present invention also proposes an electronic device, please refer to [link to relevant documentation]. Figure 3 The image shown is an electronic device according to Embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, it implements the endoscopic image processing method as described above.

[0048] In some embodiments, the processor 10 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 20 or process data, such as executing access restriction programs.

[0049] The memory 20 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 20 can be an internal storage unit of an electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 can also be an external storage device of the electronic device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, FlashCard, etc., equipped on the electronic device. Furthermore, the memory 20 can include both internal and external storage units of the electronic device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or will be output.

[0050] It should be pointed out that, Figure 3 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0051] This invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the endoscopic image processing method described above.

[0052] Those skilled in the art will understand that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0053] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0054] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0055] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0056] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. An endoscopic image processing method, characterized in that, The method includes: Acquire the surgeon's EEG and eye movement data, and align the timestamps of the EEG data, the eye movement data, and the endoscopic video stream. Based on the electroencephalogram (EEG) data and the eye movement data, neural features are extracted, including EEG features extracted from the EEG data and eye movement features extracted from the eye movement data. The neural features are input into a pre-trained radial basis function neural network, and the output is adjusted. A reinforcement learning optimizer is used to ensure that the output of the radial basis function neural network is optimal. The reinforcement learning optimizer is implemented by the DDPG algorithm. The reward function of the reinforcement learning optimizer consists of fatigue relief reward, attention optimization reward and operation fluency reward. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the eye movement features. The endoscopic images in the endoscopic video stream are adjusted according to the adjustment parameters.

2. The endoscopic image processing method according to claim 1, characterized in that, The EEG data was extracted based on occipital and parietal electrode groups, and the EEG features included alpha band relative power, theta band energy entropy, P300 wave mean amplitude, alpha band asymmetry index, beta band event-related desynchronization value, and cross-electrode coherence. The eye movement data was extracted based on fixation and saccade patterns, and the eye movement features included mean fixation duration, saccade amplitude standard deviation, red zone fixation percentage, pupil diameter variation coefficient, attention center shift, and fixation point density peak.

3. The endoscopic image processing method according to claim 2, characterized in that, The adjustment parameters include red zone suppression intensity, tissue edge enhancement coefficient, dynamic range expansion, local contrast gain, color compensation value, high-frequency noise filtering threshold, central region magnification coefficient, and motion blur correction intensity.

4. The endoscopic image processing method according to claim 3, characterized in that, The radial basis function neural network includes an input layer, a hidden layer, and an output layer. The input layer has 12 neurons, and the input layer and the hidden layer are fully connected. The hidden layer has 36 neurons and uses Gaussian radial basis functions as activation functions. The output layer has 8 neurons, and the output result is calculated by linear weighted summation. The output range is constrained to [-0.05, 0.05] by linear mapping.

5. The endoscopic image processing method according to claim 4, characterized in that, The state variables in the state space of the reinforcement learning optimizer include neural features, image features, and operational features. The image features include the proportion of red area pixels, average image brightness, high-frequency edge energy, dynamic range coverage, and color deviation index. The operational features include instrument movement speed, surgical stage labels, and historical adjustment feedback. The action variables in the action space of the reinforcement learning optimizer include the red zone suppression intensity adjustment, tissue edge enhancement coefficient adjustment, dynamic range expansion amplitude adjustment, local contrast gain adjustment, color compensation value adjustment, high-frequency noise filtering threshold adjustment, central region magnification coefficient adjustment, and motion blur correction intensity adjustment.

6. The endoscopic image processing method according to claim 5, characterized in that, The network structure of the DDPG algorithm includes an Actor network and a Critic network. The experience replay pool has a capacity of 5×10^4 experiences, and 32 batches of training are randomly sampled. The Actor network is used to input state variables and output action variables, while the Critic network is used to input state variables and action variables and output Q-values. The MSE loss function is used.

7. The endoscopic image processing method according to claim 6, characterized in that, The expression for the reward function is: R_immediate = 0.3×R_fatigue + 0.4×R_attention + 0.3×R_operation; R_fatigue = 1 - FI; FI = 0.6 × (α-wave power / (α-wave power + β-wave power)) + 0.4 × θ-band energy entropy; R_operation = 1 - Equipment movement jitter rate; R_immediate is the reward function; R_fatigue is the fatigue relief reward; FI is the fatigue index, ranging from [0,1]; α-wave power and β-wave power are extracted based on the occipital and parietal electrode groups. R_attention is the attention optimization reward, which is related to eye-tracking data. The attention optimization reward is the sum of a first reward value, a second reward value, and a third reward value. The first reward value is determined when the fixation point falls on a key area marked before surgery. The second reward value is determined when the attention entropy value is greater than a threshold. The third reward value is determined when the average fixation duration is within a preset range. The eye-tracking data includes fixation point coordinates, duration, saccade amplitude, and velocity. A Gaussian kernel function is used to transform the fixation point distribution into an attention heatmap. The weights are positively correlated with fixation duration. The attention entropy value is obtained by calculating the pixel value distribution of the attention heatmap. R_operation is the reward for smooth operation, and the device motion jitter rate is the device trajectory fluctuation rate calculated based on optical flow.

8. An endoscopic image processing system, characterized in that, For implementing the endoscopic image processing method as described in any one of claims 1-7, the system comprises: The acquisition module is used to acquire the surgeon's EEG data and eye movement data, and to align the timestamps of the EEG data, the eye movement data, and the endoscopic video stream. An extraction module is used to extract neural features based on the electroencephalogram (EEG) data and the eye movement data, wherein the neural features include EEG features extracted from the EEG data and eye movement features extracted from the eye movement data; The input module is used to input the neural features into a pre-trained radial basis function neural network and output adjustment parameters. A reinforcement learning optimizer is used to ensure that the output of the radial basis function neural network is optimal. The reinforcement learning optimizer is implemented by the DDPG algorithm. The reward function of the reinforcement learning optimizer consists of fatigue relief reward, attention optimization reward and operation fluency reward. The fatigue relief reward is determined based on the EEG features, and the attention optimization reward is determined based on the eye movement features. An adjustment module is used to adjust the endoscopic image in the endoscopic video stream according to the adjustment parameters.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the endoscopic image processing method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the endoscopic image processing method as described in any one of claims 1-7.

Images