Endoscope rotation image processing method and endoscope
By solving the endoscope rotation angle and iteratively updating the convolutional attention mechanism, the problems of endoscope image rotation and blurring were solved, achieving stable image display and high-quality image restoration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI CHUANGCHEN BIOTECHNOLOGY CO LTD
- Filing Date
- 2026-04-24
- Publication Date
- 2026-05-26
Smart Images
Figure CN122089840A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, specifically to an image processing method for medical devices, and more specifically to an endoscope rotation image processing method and an endoscope. Background Technology
[0002] Currently, in medical endoscopic diagnosis and treatment, doctors often need to rotate the endoscope to observe tissue structures from different angles. However, as the endoscope rotates, the image captured by the imaging lens also rotates, causing the displayed image to not always remain upright, increasing the difficulty for doctors to identify anatomical locations. At the same time, rotational movement can easily cause image blurring, affecting diagnostic accuracy.
[0003] While existing technologies employ image rotation correction methods using sensor data, they often focus solely on geometric rotation correction, neglecting the impact of rotation speed variations on image quality. This results in insufficient sharpness even after correction. Furthermore, conventional feature extraction networks typically utilize static convolutional kernels, lacking adaptability to dynamic rotation scenarios and hindering effective feature extraction to support subsequent high-quality image restoration.
[0004] Therefore, there is an urgent need for an endoscopic image processing method that can simultaneously achieve image straightening and high-quality deblurring. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides an endoscopic rotation image processing method and an endoscope. By actively rotating and correcting distortion and restoring blur, stable display and high-quality restoration of endoscopic images are achieved. To achieve the above objectives, the technical solution adopted by this invention is as follows:
[0006] A method for processing endoscopic rotation images is provided. The method includes: acquiring a real-time image between a previous control node and a current control node, and the rotation angle of the endoscope between the previous control node and the current control node; updating the current spatial position of the real-time image based on the rotation angle to obtain a rotated image; determining the rotation angle distribution of the endoscope; determining the parameters of a feature extraction module based on the rotation angle distribution; updating an initial feature extraction module based on the feature extraction module parameters to obtain a target feature extraction module; acquiring image features of the rotated image based on the target feature extraction module; performing fuzz estimation on the rotated image based on the image features to obtain an estimated fuzzy kernel and a current image estimate; updating the current image estimate based on a regularization term of a convolutional attention mechanism to obtain an updated image estimate value; repeating the process until a preset iteration condition is met to obtain a target image.
[0007] In some specific implementations, obtaining the rotation angle of the endoscope between the previous control node and the current control node includes: obtaining the motion data of the endoscope, decomposing the motion data to obtain the overall attitude state of the endoscope, and decomposing the rotation component in the overall attitude state to obtain the rotation angle; the endoscope data includes triaxial acceleration data and triaxial acceleration data, and the overall attitude state is based on quaternion representation.
[0008] In some specific implementations, the motion data is decomposed to obtain the overall attitude state of the endoscope, including: integrating the three-axis angular velocity data, constructing a quaternion representation based on the integration result, and updating the endoscope attitude through the quaternion representation to obtain a first endoscope attitude; obtaining the gravitational component of the three-axis acceleration, obtaining the gravity error vector of the gravity component through projection into the world coordinate system, updating the three-axis angular velocity according to the gravity error vector through an error compensation coefficient to obtain a three-axis angular velocity compensation amount, and updating the first endoscope attitude based on the three-axis angular velocity compensation amount to obtain the overall attitude state.
[0009] In some specific implementations, decomposing the rotation component in the overall attitude state to obtain the rotation angle includes: mapping and transforming the overall attitude state with the spatial relationship of the helical axis to obtain a spatial attitude representation, and separating the rotation angle in the spatial attitude representation based on the helical displacement method to obtain the rotation angle.
[0010] In some specific implementations, the feature extraction module includes an adaptive average pooling layer, a first convolutional layer, a multi-scale convolutional module, a second convolutional layer, and a dynamic convolutional layer; the parameters of the feature extraction module are the real-time weights of the dynamic convolution; determining the rotation angle distribution of the endoscope, determining the parameters of the feature extraction module based on the rotation angle distribution, and updating the initial feature extraction module based on the parameters of the feature extraction module to obtain the target feature extraction module, including: determining the degree of rotation change of the endoscope, determining the real-time weights of the dynamic convolution based on the degree of rotation change, multiplying the real-time weights with pre-set learnable convolutional parameters to obtain a real-time dynamic convolutional layer, and constructing the target feature extraction module based on the real-time dynamic convolutional layer.
[0011] In some specific implementations, obtaining image features of the rotated image based on the target feature extraction module includes: processing the rotated image through the adaptive average pooling layer and the first convolutional layer to obtain initial features, dividing the initial features into three sub-features, inputting the three sub-features into three convolutional layers of different scales in the multi-scale convolutional module, concatenating them through the multi-scale convolutional module to obtain concatenated features, and then passing the concatenated features through the second convolutional layer and the real-time dynamic convolutional layer to obtain the image features.
[0012] In some specific implementations, the convolution in the convolutional attention mechanism includes Gabor convolution, and the attention mechanism includes spatial attention and channel attention; updating the current image estimate based on the regularization term of the convolutional attention mechanism includes: updating the current image estimate multiple times through the convolutional attention mechanism, and configuring regularization update constraints during the update process.
[0013] In some specific implementations, the current image estimate is iteratively updated multiple times using a convolutional attention mechanism, including: performing convolution processing on the current image estimate based on Gabor convolution to obtain a feature map of the image features; inputting the feature map into the attention module after performing a nonlinear transformation based on the radial basis function; performing a weighted update on the feature map based on the spatial attention and the channel attention; processing the weighted updated feature map through conjugate Gabor convolution to obtain output features; mapping the output features to the image space to complete one update to obtain an updated image estimate; and performing the above process multiple times based on the updated image estimate until a preset iteration condition is reached.
[0014] Secondly, an endoscope is provided, comprising: a data acquisition terminal equipped with an IMU sensor for acquiring real-time IMU data of the endoscope, the real-time IMU data including triaxial angular velocity and triaxial acceleration; an image acquisition terminal for acquiring real-time images; and a server connected to the data acquisition terminal and the image acquisition terminal respectively, for acquiring the real-time IMU data and the real-time images, and executing any of the above-described endoscope rotation image processing methods.
[0015] In some specific implementations, the server includes: a rotation device for determining the rotation angle of the endoscope and updating the current spatial position of the real-time image based on the rotation angle to obtain a rotated image; an image processing device for updating an initial feature extraction module based on the parameters of the feature extraction module to obtain a target feature extraction module, and acquiring image features of the rotated image based on the target feature extraction module; and an image restoration device for acquiring a current image estimate and updating the current image estimate based on the regularization term of the convolutional attention mechanism to obtain an updated image estimate value, repeating the process until a preset iteration condition is met to obtain the target image.
[0016] The technical solution provided in this application updates the spatial position of the real-time image to obtain a rotated image, solving the image rotation problem caused by endoscope rotation and achieving stable image display. By determining the rotation angle distribution, the parameters of the feature extraction module are dynamically adjusted, enabling the feature extraction process to adapt to the degree of endoscope rotation, improving the accuracy and comprehensiveness of feature representation. Furthermore, a regularization term based on a convolutional attention mechanism iteratively updates the image estimation, using Gabor convolution to capture spatial frequency information and combining spatial and channel attention mechanisms to focus on key regions, effectively removing image blur and improving image restoration quality. In addition, the attitude calculation method based on quaternions and helical displacement can accurately separate the pure rotation angle from the composite motion, ensuring the accuracy of rotation correction. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] The methods, systems, and / or procedures shown in the accompanying drawings will be further described with reference to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example figures represent similar mechanisms in the various views of the drawings.
[0019] Figure 1 This is a schematic diagram of the endoscope system structure provided in the embodiments of this application.
[0020] Figure 2 This is a schematic flowchart of the endoscope rotation image processing method provided in the embodiments of this application.
[0021] Figure 3 This is a schematic diagram of the server structure provided in an embodiment of this application.
[0022] Figure 4 The image is a real-time image to be processed.
[0023] Figure 5 This is the processed target image.
[0024] Figure 6 This is a schematic diagram of the terminal device structure provided in the embodiments of this application. Detailed Implementation
[0025] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.
[0026] In the detailed description below, numerous specific details are illustrated with examples to provide a comprehensive understanding of the relevant guidance. However, it will be apparent to those skilled in the art that this application can be practiced without these details. In other instances, well-known methods, procedures, systems, components, and / or circuits have been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of this application.
[0027] This application uses flowcharts to illustrate the execution process performed by a system according to embodiments of this application. It should be clearly understood that the execution processes in the flowcharts may not be executed sequentially. Instead, these execution processes may be executed in reverse order or simultaneously. Additionally, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.
[0028] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.
[0029] (1) In response to, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.
[0030] (2) Based on, used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order of execution of the multiple operations.
[0031] Medical endoscopes are effective medical devices for diagnosing digestive tract diseases. Because they do not involve radiation damage and can accurately observe pathological changes in internal organs, they are widely used for the examination, diagnosis, and treatment of the digestive system, including the esophagus, stomach, and intestines. To maintain a compact overall structure, the image acquisition unit is integrated with the endoscope body, avoiding the problem of a large endoscope size caused by a connection structure between the image acquisition unit and the endoscope body. However, this design causes the image acquisition unit and the endoscope body to rotate synchronously, resulting in the acquired image rotating synchronously with the image acquisition unit, further complicating the doctor's use.
[0032] Furthermore, the movement of the endoscope is directly affected by gastrointestinal peristalsis. When the speed of gastrointestinal peristalsis increases, the acquired images will be blurry due to the peristalsis.
[0033] Therefore, in order to solve the problems of image rotation display and image blurring in the prior art, this application provides an endoscope. For details on the automated control part of this endoscope, please refer to... Figure 1 The endoscope includes a main body, a data acquisition terminal 11 and an image acquisition terminal 12 disposed on the main body, and a server 13 that communicates with the data acquisition terminal and the image acquisition terminal.
[0034] The data acquisition terminal is equipped with an IMU sensor to acquire real-time IMU data from the endoscope, including triaxial angular velocity and triaxial acceleration. Specifically, the IMU sensor is typically integrated inside the endoscope's handle or at the front end of the endoscope body, enabling real-time sensing of the endoscope's motion in three-dimensional space at a high sampling frequency. The triaxial angular velocity data reflects the speed of the endoscope's rotation around each axis, while the triaxial acceleration data reflects the endoscope's translational motion and the direction of gravity. This raw motion data is transmitted in real-time to the server via data transmission lines, providing the basic data source for subsequent attitude calculations. It should be understood that the specific model and installation location of the IMU sensor can be adapted to the endoscope's model and structure, as long as it can accurately acquire motion data.
[0035] The image acquisition terminal is used to acquire real-time images. The image sensor, located at the front end of the endoscope, converts the acquired optical signals into digital image signals, which constitute the real-time image. The image acquisition terminal transmits these real-time images to a server via a high-definition cable or wireless transmission module. During the doctor's endoscope examination, the image acquisition terminal continuously operates, generating a series of real-time image frames.
[0036] The server connects to both the data acquisition terminal and the image acquisition terminal to acquire real-time IMU data and real-time images. Furthermore, an executable program is configured within the server, constituting a method for processing the acquired rotating images. This method is described as an endoscopic rotating image processing method; please refer to [link to relevant documentation] for more information. Figure 2 This includes the following steps:
[0037] Step S21. Obtain the real-time image between the previous control node and the current control node, as well as the rotation angle of the endoscope between the previous control node and the current control node, and update the current spatial position of the real-time image based on the rotation angle to obtain the rotational image.
[0038] In this embodiment, a control node refers to a trigger moment or position marker used to determine the rotation cycle of the endoscope. In actual operation, when a doctor holds the endoscope for examination, they often need to rotate the endoscope to observe tissues at different angles. The time interval between the previous control node and the current control node corresponds to one complete rotation operation. During this process, the image acquisition terminal continuously acquires real-time images, while the data acquisition terminal records the motion data of the endoscope. By calculating the motion data within this time interval, the rotation angle of the endoscope relative to the previous posture can be obtained. After obtaining the rotation angle, the system uses an image transformation algorithm to perform a reverse transformation on the pixel coordinates of the real-time image, restoring the rotated image to the forward viewing angle, thus obtaining a rotated image. This step ensures that the displayed image always matches the doctor's observation habits, avoiding recognition difficulties caused by image rotation.
[0039] Specifically, the rotation angle is obtained by acquiring the motion data of the endoscope, decomposing the motion data to obtain the overall posture state of the endoscope, and decomposing the rotation component in the overall posture state to obtain the rotation angle.
[0040] The endoscopic data includes triaxial acceleration data and triaxial angular velocity data, and the overall attitude state is represented by quaternions.
[0041] Specifically, motion data is acquired by an inertial measurement unit (IMU) mounted on the endoscope. This IMU sensor includes a three-axis gyroscope and a three-axis accelerometer, capable of outputting three-axis angular velocity data and three-axis acceleration data, respectively. Because the endoscope's attitude changes complexly during movement within the body cavity, potentially involving large-angle rotations, using traditional Euler angles to represent attitude can lead to gimbal lock when the pitch angle reaches ±90 degrees, causing attitude calculation failure. Therefore, this embodiment uses quaternions to represent the overall attitude state of the endoscope. Quaternions not only avoid the gimbal lock problem but also offer advantages such as low computational complexity and smooth interpolation, making them more suitable for real-time attitude calculation scenarios.
[0042] Because IMU data has a high frequency, it can complement low-frequency, high-precision visual data effectively. However, due to the inherent steady-state error of the IMU and its susceptibility to environmental noise, the relative pose obtained through integration will accumulate errors. To minimize these errors and obtain more accurate IMU pose data, this invention first preprocesses the IMU data, then calculates the IMU position and orientation data, taking noise into account during the process, and finally obtains usable IMU pose data.
[0043] The main purpose of IMU data preprocessing is to filter out noise and restore the IMU measurement data to obtain results that are closer to the true values. IMU measurement data can be decomposed into four parts: true value, zero bias, white noise, and gravity component. The corresponding decompositions of angular velocity and acceleration are expressed based on the following formulas: , ; , Where ωtrue and 𝑎true are the true angular velocity and acceleration, respectively; 𝑏𝑔(𝑡) and 𝑏𝑎(𝑡) are the time-varying zero biases of angular velocity and acceleration, respectively; 𝑛𝑔(𝑡) and 𝑛𝑎(𝑡) are the time-varying white noises of angular velocity and acceleration, respectively; 𝑅(𝑞) are the rotation matrices corresponding to the IMU attitude quaternions; 𝑅T(𝑞)𝑔 are the projections of gravitational acceleration 𝑔 in the IMU coordinate system.
[0044] Among them, for and Follows a random walk process: , ; , .
[0045] In this invention, considering the above-mentioned characteristics of the IMU, the processing mainly involves compensating for the zero bias and using a filter to remove high-frequency noise from the accelerometer, while also eliminating outliers. The IMU's zero bias processing is specifically compensated by recording the zero bias value under the initial attitude of the endoscope. The zero bias value under the initial attitude is shown in the following formula: , ;in The initial attitude of the IMU is represented by N, which is the number of data samples taken during the initial attitude holding time. In this invention, it is set to 50.
[0046] To reduce high-frequency noise from the accelerometer, mean filtering is used. The acceleration data after mean filtering is as follows: , where W is the size of the mean filter.
[0047] Finally, to avoid abrupt changes in accelerometer data, this invention uses Mahalanobis distance to detect outliers in the acceleration data, removes these outliers, and then fills them in using linear interpolation of the left and right data. The Mahalanobis distance is calculated based on the following formula: ,in This represents the average acceleration data within the window. This represents the covariance of the acceleration data within the window.
[0048] In this embodiment, the acquired triaxial angular velocity data is integrated, and a quaternion representation is constructed based on the integration result. This quaternion representation is then used to update the endoscope's attitude, resulting in the first endoscope attitude. The quaternion construction is achieved using IMU data that has undergone noise reduction processing. The core purpose of this construction is to fuse angular velocity and acceleration to obtain a quaternion that comprehensively reflects the endoscope's motion state.
[0049] In this embodiment, the quaternion construction process is implemented using complementary filtering. By combining the high-frequency dynamic response capability of gyroscope data and the stability capability of accelerometer data in the direction of gravity, uniform attitude construction is achieved.
[0050] Specifically, this process begins by integrating the angular velocity. The angular velocity, after processing in the previous step, is expressed as follows: The IMU attitude is updated using quaternion differential equations, which are based on the following expression: ,in The sampling time interval of the gyroscope. It is an exponential mapping of quaternions.
[0051] Then, the accelerometer is corrected for the direction of gravity. Specifically, since the endoscope typically moves at a constant speed, the linear motion component is extracted in this step to obtain the gravity component of the accelerometer. Then, projection processing into the world coordinate system is performed based on the following formula: Then, based on the calculated accelerometer gravity error vector, a proportional-integral error compensation coefficient is designed to correct the angular velocity of the acceleration data. The proportional-integral error step size coefficient is calculated based on the following formula: .
[0052] The proportional term is used to adjust the response speed of the accelerometer calibration, but if it is too large, it will amplify high-frequency noise and cause attitude data oscillation. The integral term is used to suppress the zero-bias drift of the accelerometer, but if it is too large, it will cause instantaneous interference and instantaneous numerical saturation. This application determines whether the endoscope is in violent motion by detecting the variance of the acceleration count value, and limits the integral term by ±0.1 sd / s to deal with the integral saturation phenomenon.
[0053] The above steps yielded the IMU attitude quaternion from the gyroscope integration and the accelerometer acceleration compensation term. These data were then fused to obtain the updated attitude quaternion, which is represented by the following formula: After updating, the values are normalized to obtain the normalized IMU pose quaternions.
[0054] Furthermore, after obtaining the attitude results, the rotation angle of the endoscope needs to be analyzed. Rotation representations typically include Euler angles, quaternions, rotation matrices, and helical axis representations. These methods accurately describe the rotational state of an object relative to a reference coordinate system and can be converted between each other during attitude calculation and description. This embodiment uses unit quaternions to describe the calculated attitude, achieves the attitude representation of the moving body through the conversion between quaternions and helical axis representations, and separates the pure rotation angles in the composite motion using the helical displacement method.
[0055] Specifically, endoscopy often involves a combination of translation and rotation during operation, while image straightening only requires the pure rotation component. Using attitude matrices or Euler angles directly makes it difficult to intuitively separate the pure rotation angle around a specific axis. This embodiment introduces the helical axis theory to map the calculated overall attitude state into a spatial attitude representation. Based on the helical displacement method, the motion of any rigid body can be viewed as a combination of translation along a helical axis and rotation around that axis. Through mathematical decomposition, the rotation angle in the spatial attitude representation can be accurately separated. This step eliminates the interference of translational motion on angle calculation, ensuring that the final obtained rotation angle reflects only the rotational movement of the endoscope body, thus providing accurate geometric parameters for subsequent image straightening.
[0056] The conversion relationship between quaternions and the spiral axis representation is expressed by the following formula: , Where a, b, c, and d represent four real numbers, and i, j, and k represent three imaginary units, and , It is an axis vector.
[0057] Step S22. Determine the rotation angle distribution of the endoscope, determine the parameters of the feature extraction module based on the rotation angle distribution, update the initial feature extraction module based on the feature extraction module parameters to obtain the target feature extraction module, and obtain the image features of the rotated image based on the target feature extraction module.
[0058] Specifically, the rotation angle distribution reflects the dynamic characteristics of the endoscope during rotation, such as changes in rotation speed. This embodiment innovatively proposes adaptively adjusting the parameters of the feature extraction module based on these dynamic rotation characteristics. This is because the degree of endoscope rotation directly affects the degree of image blur: the faster the rotation speed, the more severe the motion blur, requiring the feature extraction module to have stronger feature capture capabilities; conversely, when the rotation is gentle, the image quality is relatively good, and the feature extraction module can use more general parameter settings. By analyzing the rotation angle distribution, the system can generate feature extraction module parameters adapted to the current motion state, thereby updating the parameters of the initial feature extraction module and constructing a target feature extraction module optimized for the current scene. Subsequently, the rotated image obtained in step S21 is input into this target feature extraction module to extract key image features. This adaptive mechanism effectively improves the targeting and effectiveness of feature extraction, providing high-quality data support for subsequent image restoration.
[0059] In this embodiment, the feature extraction module includes an adaptive average pooling layer, a first convolutional layer, a multi-scale convolutional module, a second convolutional layer, and a dynamic convolutional layer. The parameters of the feature extraction module are the real-time weights of the dynamic convolution; adjusting the parameters of the feature extraction module is essentially adjusting the real-time weights of the dynamic convolution.
[0060] In image processing, conventional feature extraction typically employs static convolutional networks, where the weights of the convolutional kernels remain fixed after training. However, in endoscopic examinations, the rotation speed and amplitude of the endoscope are highly random. When rotation is rapid, the image experiences severe motion blur, requiring the feature extraction module to possess stronger edge and texture capture capabilities; conversely, when rotation is gradual, the image remains relatively clear, necessitating a module focused on preserving detailed information. Static convolutional networks cannot perceive these dynamic changes in the input image, limiting their feature representation capabilities when processing rotated and blurred images, making it difficult to simultaneously handle both clear and blurred states.
[0061] To address the aforementioned issues, this embodiment proposes an adaptive feature extraction mechanism. Specifically, the degree of rotation of the endoscope is first determined, which can be quantified by the rate of change of the rotation angle over time. Then, the real-time weights of the dynamic convolution are determined based on this degree of rotation. These real-time weights are multiplied by pre-set learnable convolution parameters to obtain a real-time dynamic convolutional layer, and a target feature extraction module is constructed based on this layer. In this way, the feature extraction module can adjust its receptive field and feature selection strategy in real time according to the endoscope's motion state, effectively extracting blurred features during rapid rotation and preserving high-frequency details during gentle rotation, significantly improving the model's robustness to input images of varying quality.
[0062] Furthermore, the process of obtaining image features of the rotated image based on the target feature extraction module specifically includes: the rotated image is first processed through an adaptive average pooling layer and a first convolutional layer to obtain initial features. Then, the initial features are divided into three sub-features. This division operation allows the model to process information in parallel from multiple branches. The three sub-features are respectively input into three convolutional layers of different scales in the multi-scale convolutional module; for example, convolutional kernels of 1x1, 3x3, and 5x5 sizes can be used. Different sized convolutional kernels have different receptive fields; small-scale convolutional kernels are good at capturing detailed textures, while large-scale convolutional kernels are good at capturing global structure and contextual information. After processing by the multi-scale convolutional module, the outputs of the three branches are concatenated to obtain concatenated features, thus fusing multi-scale feature information. Finally, the concatenated features are processed through a second convolutional layer and a real-time dynamic convolutional layer to obtain the final image features. This split-concat structure not only enriches the diversity of features, but also adaptively weights multi-scale features through subsequent dynamic convolutional layers, ensuring that the output features contain rich details and have the best ability to express the current rotation state.
[0063] Step S23. Perform blur estimation on the rotated image based on image features to obtain the estimated blur kernel of the rotated image and the current image estimate. Update the current image estimate based on the regularization term of the convolutional attention mechanism to obtain the updated image estimate value. Repeat the process until the preset iteration condition is met to obtain the target image.
[0064] Specifically, after acquiring image features, the system first estimates the blur level of the rotated image, calculates the blur kernel causing the blurring, and provides a preliminary current image estimate. Then, it performs image restoration based on this estimate. In this embodiment, the blur estimation is implemented using L0 regularization, and the image restoration process employs iterative optimization, with each iteration updating the current image estimate.
[0065] Furthermore, in each iteration of image restoration, this embodiment introduces a regularization term from the convolutional attention mechanism as a constraint. This regularization term utilizes convolution operations to extract deep features of the image and combines them with the attention mechanism to focus on key areas of the image, such as texture details and edge contours, while suppressing background noise and irrelevant information. Through this regularization constraint, the current image estimate can be corrected to obtain an updated image estimate value that is closer to the real scene. The system continuously repeats the above estimation and update process until the preset iteration conditions are met. The final output target image is a high-quality, clear image after rotation and deblurring. Through the above steps, this embodiment realizes a complete process from image rotation correction to adaptive feature extraction to iterative image restoration, significantly improving the visual quality and diagnostic value of endoscopic images.
[0066] In this process, regularization update constraints are applied to each iteration using a convolutional attention mechanism. Gabor convolution is used for the convolution, and the attention mechanism includes spatial attention and channel attention.
[0067] Specifically, at the beginning of each iteration, the current image estimate is input into a pre-defined Gabor convolution module. This module contains multiple sets of Gabor filters of different scales and orientations, capable of capturing frequency information from different dimensions to obtain feature maps. For example, for vascular textures or tissue boundaries in endoscopic images, Gabor filters can extract their directional features, forming feature maps containing rich high-frequency information. This step provides the basic data support for subsequent feature enhancement. The feature maps are then nonlinearly transformed based on radial basis functions and input into the attention module.
[0068] To enhance the expressive power of the features, this embodiment introduces radial basis functions (RBFs) to perform nonlinear transformations on the extracted feature maps. RBF networks possess strong nonlinear mapping capabilities, enabling them to map features from low-dimensional space to high-dimensional space, thus making originally linearly inseparable features separable and further extracting deeper information from the feature maps. The feature maps after nonlinear transformation are then fed into the attention module, preparing for subsequent weighted processing.
[0069] In this embodiment, the attention module performs weighted updates to the feature map using spatial attention and channel attention. Specifically, the attention module is a key component in improving image restoration quality. Endoscopic images often contain significant background noise and critical diagnostic regions. The spatial attention mechanism learns a weight map, assigning different weights to each spatial location in the image, thereby guiding the model to focus on severely blurred areas or regions containing crucial textures, while suppressing background noise interference. The channel attention mechanism, on the other hand, weights multiple feature channels generated by Gabor convolution, strengthening channel features that contribute significantly to the deblurring task and suppressing redundant or irrelevant channels. Through the synergistic effect of spatial and channel attention, the model can adaptively select high-value features, improving the targeted nature of iterative updates.
[0070] The weighted updated feature map is processed by conjugate Gabor convolution to obtain the output features. The output features are then mapped to the image space to complete one update and obtain the updated image estimate.
[0071] Specifically, the attention-weighted feature map contains enhanced key information, but it is still in the feature space. To restore it to information in the image space, the system uses conjugate Gabor convolution for inverse mapping. Conjugate Gabor convolution and forward Gabor convolution are conjugate in frequency response, effectively reconstructing information from the feature domain back to the pixel domain, thus obtaining the image residual or correction amount for this iteration. This correction amount is then superimposed on the current image estimate, completing an image estimation update and obtaining the updated image estimate.
[0072] The above process is repeated multiple times to complete multiple iterations until the preset iteration conditions are met, thereby completing the image update and obtaining the target image.
[0073] In this embodiment, the iteration condition can be either that the number of iterations has reached a preset maximum number of iterations, or that the difference between two adjacent updated image estimates is less than a preset convergence threshold. If the condition is not met, the updated image estimate is used as the new current image estimate, and the iteration continues for optimization; if the condition is met, the current image estimate is output as the final target image. Through this iterative update based on the convolutional attention mechanism regularization term, this embodiment can gradually remove motion blur caused by rotation while preserving the edge details of the image, significantly improving the clarity and recognizability of endoscopic images.
[0074] In summary, this application provides a method for processing rotated endoscopic images. It updates the spatial position of real-time images to obtain a rotated image, solving the image rotation problem caused by endoscope rotation and achieving stable image display. By dynamically adjusting the parameters of the feature extraction module based on the determined rotation angle distribution, the feature extraction process can adapt to the degree of rotation of the endoscope, improving the accuracy and comprehensiveness of feature representation. Furthermore, a regularization term based on a convolutional attention mechanism iteratively updates the image estimation, using Gabor convolution to capture spatial frequency information and combining spatial and channel attention mechanisms to focus on key regions, effectively removing image blur and improving image restoration quality. In addition, the attitude calculation method based on quaternions and helical displacement can accurately separate the pure rotation angle from the composite motion, ensuring the accuracy of rotation correction.
[0075] For further details, please refer to [link / reference]. Figure 3 This is a schematic diagram of the internal logical structure of the server. Specifically, the server 13 includes a straightening device 131, an image processing device 132, and an image restoration device 133 in terms of logical functions. This division allows the server to modularly execute the image processing methods described in steps S21-S23.
[0076] Specifically, the rotation device is used to determine the rotation angle of the endoscope and update the current spatial position of the real-time image based on the rotation angle to obtain a rotated image. This device mainly corresponds to the attitude calculation process in step S21. In terms of hardware logic, the rotation device is responsible for receiving raw IMU data from the data acquisition terminal, performing calculations such as quaternion integration, error compensation, and helical displacement method to separate the rotation angle, and performing geometric transformations on the real-time image transmitted from the image acquisition terminal accordingly. Encapsulating the rotation function independently as a rotation device is beneficial for developers to independently optimize and debug the attitude calculation algorithm. For example, when it is necessary to change the attitude fusion algorithm, there is no need to modify the image recovery-related code, thereby reducing system maintenance costs.
[0077] The image processing device is used to update the initial feature extraction module based on the parameters of the feature extraction module to obtain the target feature extraction module, and to acquire the image features of the rotated image based on the target feature extraction module. This device mainly corresponds to the adaptive feature extraction process detailed in step S22. The image processing device receives the rotated image output by the rotating device and dynamically generates convolution weights according to the rotation angle distribution results to construct a target feature extraction module adapted to the current motion state. By making the feature extraction logic an independent image processing device, the server can flexibly configure the hierarchical structure of the feature extraction network, for example, loading convolutional networks of different depths according to different endoscope models or application scenarios, without interfering with the front-end rotating logic or the back-end recovery logic.
[0078] The image restoration device acquires the current image estimate and updates it based on the regularization term of the convolutional attention mechanism, obtaining the updated image estimate value. This process is repeated until a preset iteration condition is met to obtain the target image. This device mainly corresponds to the iterative restoration process detailed in step S23. The image restoration device uses Gabor convolution and attention mechanisms to gradually remove image blur. Setting the restoration logic independently facilitates dynamic scheduling of server computing resources. For example, in a multi-threaded processing environment, more GPU computing resources can be allocated to the image restoration device to accelerate the iterative convergence process.
[0079] It should be understood that the above-mentioned division of the straightening device, image processing device, and image restoration device is a logical functional division. In terms of physical implementation, they can be integrated on the same processor (such as a CPU, GPU, or embedded chip), with the processor executing different software program modules to achieve their respective functions; or they can be deployed on different physical hardware. For example, the straightening device can be deployed on an FPGA chip inside the endoscope handle for low-latency preprocessing, while the image processing device and image restoration device can be deployed on a cloud server to utilize powerful computing power for deep learning inference. This modular architecture design not only improves the maintainability and scalability of the system but also enables the system to adapt to different deployment requirements, from local processing to cloud processing.
[0080] Furthermore, this embodiment, in conjunction with a specific endoscope application scenario, provides a detailed explanation of the endoscopic rotation image processing method and the application process of the endoscope provided in the aforementioned embodiment. Specifically, this scenario involves a doctor performing an intestinal examination on a patient using a handheld endoscope. During the examination, the doctor often needs to rotate the endoscope to observe lesions on the intestinal wall or within the folds, which causes rotation and motion blur in the acquired images.
[0081] Specifically, the doctor inserts the endoscope into the patient's intestines. When it reaches the predetermined position or when a rotational observation operation needs to begin, the doctor triggers a control node located on the endoscope handle, such as pressing a physical button. The system records this moment as the previous control node. Subsequently, the doctor begins to rotate the endoscope to adjust the observation angle. During this process, the image acquisition terminal continuously acquires real-time images, while the IMU sensor inside the endoscope acquires triaxial angular velocity and triaxial acceleration data at high frequency in real time.
[0082] When the rotation operation ends, the doctor triggers the control node again, and the system records this moment as the current control node. The server then acquires the real-time images and IMU data between the previous and current control nodes. The rotation correction device inside the server first processes the IMU data: by integrating the three-axis angular velocities to construct a quaternion representation, and using the gravitational component of the three-axis acceleration for error compensation, the overall attitude state of the endoscope is calculated; then, the pure rotation angle is separated from this overall attitude state using the helical displacement method. After obtaining the rotation angle, the rotation correction device updates the spatial position of the real-time image based on this angle, thereby correcting the intestinal image, which was originally tilted with the rotation of the endoscope, to a positive viewing angle, resulting in a rotated image. This process effectively solves the cognitive burden on doctors who need to perform additional image orientation correction in their brains when rotating the endoscope, ensuring that the field of view remains positive and reducing the difficulty of operation.
[0083] For the straightened image, feature extraction is performed using an image acquisition device. Since the rotation speed directly affects the degree of image blur, the system first analyzes the distribution of rotation angles and calculates the degree of rotation change. If a fast rotation speed is detected, the system determines that the image may be severely blurred and generates larger dynamic convolution weights to construct a target feature extraction module focused on capturing high-frequency texture features. If the rotation speed is slow, moderate weight parameters are generated. The straightened image is processed by this adaptively updated target feature extraction module, which extracts features from different receptive fields through a multi-scale convolution module and then weights them through a dynamically convolutional layer to obtain image features containing rich texture information. This adaptive mechanism ensures that regardless of the speed of the doctor's operation, the system can extract the key features most beneficial for subsequent recovery.
[0084] Finally, the image restoration device iteratively restores the rotated image based on the extracted image features. The system first estimates the blur kernel and generates a preliminary current image estimate. Then, during the iteration process, it uses Gabor convolution to extract the spatial frequency information of the feature map, performs nonlinear transformation through radial basis functions, and combines spatial attention and channel attention mechanisms to focus on the lesion tissue region, suppressing background noise such as mucus and bubbles in the intestine. After multiple iterations, the system gradually removes the blur caused by the rotational motion, ultimately outputting a target image with clear texture and sharp edges, which is displayed in real-time on the doctor's monitor.
[0085] In summary, the technical solution provided by this invention, through the processing logic of "rotation detection - adaptive feature extraction - iterative recovery," not only automatically corrects the rotated image when the doctor rotates the endoscope for observation, but also adaptively restores image clarity according to the rotation speed, greatly improving the image quality of endoscopic examinations and the doctor's user experience. Please refer to... Figure 4 and Figure 5This represents the application result of this embodiment in the above scenario. Figure 4 and Figure 5 These are the real-time image to be processed and the processed target image, respectively, through... Figure 4 and Figure 5 As can be seen, the rotation image processing method provided in this embodiment can correct and deblur an image that was originally rotated and blurred, thereby achieving a complete and clear representation of the image.
[0086] See Figure 6 The above methods can also be integrated into the provided terminal device 500. Since the device may vary significantly due to differences in configuration or performance, it may include one or more processors 501 and memories 502. The memories 502 may store one or more application programs or data. The memories 502 can be temporary or persistent storage. The application programs stored in the memories 502 may include one or more modules (not shown in the figure), each module may include a series of computer-executable instructions from the terminal device. Furthermore, the processor 501 may be configured to communicate with the memories 502, and the terminal device may execute the series of computer-executable instructions stored in the memories 502. The terminal device may also include one or more power supplies 503, one or more wired / wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, etc.
[0087] In one specific embodiment, the terminal device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the terminal device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:
[0088] The real-time image between the previous control node and the current control node, as well as the rotation angle of the endoscope between the previous control node and the current control node, are obtained, and the current spatial position of the real-time image is updated based on the rotation angle to obtain the rotational image.
[0089] The rotation angle distribution of the endoscope is determined, the parameters of the feature extraction module are determined based on the rotation angle distribution, the initial feature extraction module is updated based on the feature extraction module parameters to obtain the target feature extraction module, and the image features of the rotation image are obtained based on the target feature extraction module.
[0090] Based on the image features, a blur estimation is performed on the rotated image to obtain an estimated blur kernel and a current image estimate. The current image estimate is then updated based on a regularization term of the convolutional attention mechanism to obtain an updated image estimate value. This process is repeated until a preset iteration condition is met to obtain the target image.
[0091] Optionally, the processor can perform various functions by running or executing software programs stored in memory and calling data stored in memory, such as executing the methods corresponding to steps S21-S23.
[0092] In a specific implementation, as one example, the processor may include one or more microprocessors.
[0093] The memory is used to store the software program that executes the solution of this application, and the execution is controlled by the processor. The specific implementation method can be referred to the above method embodiment, which will not be repeated here.
[0094] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0095] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0096] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0098] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for processing endoscopic rotational images, characterized in that, The method includes: The real-time image between the previous control node and the current control node, as well as the rotation angle of the endoscope between the previous control node and the current control node, are obtained, and the current spatial position of the real-time image is updated based on the rotation angle to obtain the rotational image. The rotation angle distribution of the endoscope is determined, the parameters of the feature extraction module are determined based on the rotation angle distribution, the initial feature extraction module is updated based on the feature extraction module parameters to obtain the target feature extraction module, and the image features of the rotation image are obtained based on the target feature extraction module. Based on the image features, a blur estimation is performed on the rotated image to obtain an estimated blur kernel and a current image estimate. The current image estimate is then updated based on a regularization term of the convolutional attention mechanism to obtain an updated image estimate value. This process is repeated until a preset iteration condition is met to obtain the target image.
2. The endoscopic rotation image processing method according to claim 1, characterized in that, Obtaining the rotation angle of the endoscope between the previous control node and the current control node includes: acquiring the motion data of the endoscope, decomposing the motion data to obtain the overall attitude state of the endoscope, and decomposing the rotation component in the overall attitude state to obtain the rotation angle; the motion data of the endoscope includes three-axis angular velocity data and three-axis acceleration data, and the overall attitude state is based on quaternion representation.
3. The endoscopic rotation image processing method according to claim 2, characterized in that, The motion data is decomposed to obtain the overall attitude state of the endoscope, including: integrating the three-axis angular velocity data, constructing a quaternion representation based on the integration result, and updating the endoscope attitude through the quaternion representation to obtain a first endoscope attitude; obtaining the gravitational component of the three-axis acceleration, obtaining the gravity error vector of the gravity component through projection into the world coordinate system, updating the three-axis angular velocity according to the gravity error vector through an error compensation coefficient to obtain a three-axis angular velocity compensation amount, and updating the first endoscope attitude based on the three-axis angular velocity compensation amount to obtain the overall attitude state.
4. The endoscopic rotation image processing method according to claim 3, characterized in that, The rotation component in the overall attitude state is decomposed to obtain the rotation angle, including: mapping the overall attitude state to the spatial relationship of the helical axis to obtain a spatial attitude representation, and separating the rotation angle in the spatial attitude representation based on the helical displacement method to obtain the rotation angle.
5. The endoscopic rotation image processing method according to claim 4, characterized in that, The feature extraction module includes an adaptive average pooling layer, a first convolutional layer, a multi-scale convolutional module, a second convolutional layer, and a dynamic convolutional layer. The parameters of the feature extraction module are the real-time weights of the dynamic convolutional layer. The module determines the rotation angle distribution of the endoscope, determines the parameters of the feature extraction module based on the rotation angle distribution, and updates the initial feature extraction module based on the parameters to obtain the target feature extraction module. This includes: determining the degree of rotation change of the endoscope, determining the real-time weights of the dynamic convolutional layer based on the degree of rotation change, multiplying the real-time weights by preset learnable convolutional parameters to obtain the real-time dynamic convolutional layer, and constructing the target feature extraction module based on the real-time dynamic convolutional layer.
6. The endoscopic rotation image processing method according to claim 5, characterized in that, The image features of the rotated image are obtained based on the target feature extraction module, including: the rotated image is processed by the adaptive average pooling layer and the first convolutional layer to obtain initial features, and the initial features are divided into three sub-features. The three sub-features are respectively input into three convolutional layers of different scales in the multi-scale convolutional module. The multi-scale convolutional module concatenates the sub-features to obtain concatenated features. The concatenated features are then processed by the second convolutional layer and the real-time dynamic convolutional layer to obtain the image features.
7. The endoscopic rotation image processing method according to claim 1, characterized in that, The convolutional attention mechanism includes Gabor convolution, and the attention mechanism includes spatial attention and channel attention. The current image estimate is updated using a regularization term based on a convolutional attention mechanism, including: performing multiple iterative updates on the current image estimate using a convolutional attention mechanism, and configuring regularization update constraints during the update process.
8. The endoscopic rotation image processing method according to claim 7, characterized in that, The current image estimate is iteratively updated using a convolutional attention mechanism, including: performing convolution processing on the current image estimate based on Gabor convolution to obtain a feature map of the image features; inputting the feature map into the attention module after performing a nonlinear transformation based on the radial basis function; performing a weighted update on the feature map based on the spatial attention and the channel attention; processing the weighted updated feature map through conjugate Gabor convolution to obtain output features; mapping the output features to the image space to complete one update to obtain an updated image estimate; and performing the above process iteratively multiple times based on the updated image estimate until a preset iteration condition is reached.
9. An endoscope, characterized in that, include: The data acquisition terminal is equipped with an IMU sensor to acquire real-time IMU data of the endoscope, including triaxial angular velocity and triaxial acceleration. Image acquisition terminal, used to acquire real-time images; The server is connected to the data acquisition terminal and the image acquisition terminal respectively, and is used to acquire the real-time IMU data and the real-time image, and execute the endoscopic rotation image processing method according to any one of claims 1-8.
10. The endoscope according to claim 9, characterized in that, The server includes: A rotation device is used to determine the rotation angle of the endoscope and update the current spatial position of the real-time image based on the rotation angle to obtain a rotation-corrected image; An image processing device is used to update an initial feature extraction module based on the parameters of the feature extraction module to obtain a target feature extraction module, and to obtain image features of the rotated image based on the target feature extraction module; An image restoration device is used to obtain a current image estimate and update the current image estimate based on a regularization term of a convolutional attention mechanism to obtain an updated image estimate value. This process is repeated until a preset iteration condition is met to obtain the target image.