Image returning method and device of endoscope, storage medium and terminal

Abstract

The invention discloses an image return method and device of an endoscope, a storage medium and a terminal. The method comprises the following steps: acquiring an initial image and a triaxial angular velocity signal of the endoscope; according to the triaxial angular velocity signal, attitude calculation is carried out by using a quaternion differential equation to obtain an attitude quaternion, and the attitude quaternion is updated in real time; performing interpolation on the attitude quaternions at adjacent moments within one frame of exposure time to obtain a row-level quaternion and generate a row-level rotation matrix; and based on the row-level rotation matrix, performing return and compensation on the initial image by using reverse mapping and bilinear interpolation to obtain a return image. The problems of universal joint deadlock and function failure caused by the universal joint deadlock in Euler angle attitude calculation are avoided, line-by-line compensation is carried out through an interpolation algorithm, the jelly effect is eliminated, the problems of large delay, insufficient real-time performance and difficulty in integration in a limited space in post-processing of a traditional software algorithm or an external micro-control unit are avoided, and the method is suitable for being applied to the field of Euler angle attitude calculation. And the imaging requirement of clinical operation of the medical endoscope can be better met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of endoscopy, in particular to an image back-right method, device, storage medium and terminal of an endoscope. BACKGROUND

[0002] In the field of medical endoscopy, the change of the angle and posture of the endoscope in clinical operation often leads to the problems of tilt, rotation or jitter of the collected image, which seriously affects the observation and diagnosis of the doctor on the lesion, so the posture solving and image stabilization are very important. From the perspective of posture solving technology, the traditional scheme mostly uses Euler angles to represent the spatial posture, and its core principle is to describe the posture by three successive rotations around fixed coordinate axes. However, the three-dimensional rotation space is a three-dimensional manifold, and Euler angles are represented by only three angles. When the pitch angle θ is close to ±90°, the two rotation axes will coincide, resulting in the loss of one degree of freedom of the system and the occurrence of gimbal lock, which cannot track the rapid flipping or rotating action of the endoscope, and also cannot identify the angle information, resulting in the problems of posture solving interruption and image back-right function failure. At the same time, the Euler angle algorithm needs to perform trigonometric function calculation multiple times, which is difficult to realize efficient operation in the hardware platform and lacks real-time performance.

[0003] From the perspective of image stabilization technology, the existing technology mainly performs post-processing based on software algorithm or external micro control unit, and finally displays the rotated image by obtaining the rotation angle of the gyroscope and rotating the collected image based on the rotation angle. However, the software processing has large delay and cannot meet the real-time operation requirements of the endoscope; the integration of the external micro control unit increases the power consumption of the system, and in the limited space scene such as disposable electronic endoscope, the integration of high-performance processors is extremely difficult, and it is difficult to guarantee the image stability and accuracy in clinical application.

[0004] Therefore, how to use an effective method to overcome the defects of the existing technology has become a technical problem to be solved. SUMMARY

[0005] The purpose of the present application is to provide an image back-right method, device, storage medium and terminal of an endoscope to solve the above problems.

[0006] The technical scheme of the present application is as follows: an image back-right method of an endoscope, comprising the following steps: obtaining an initial image and a three-axis angular velocity signal of the endoscope; performing posture solving on the three-axis angular velocity signal to obtain a posture quaternion by using a quaternion differential equation, and updating the posture quaternion in real time; interpolating the posture quaternions of adjacent time points to obtain a row-level quaternion within one frame exposure time, generating a row-level rotation matrix according to the row-level quaternion; and performing back-right and compensation on the initial image based on the row-level rotation matrix to obtain a back-right image.

[0007] As an improvement of the embodiment of the present application, the "obtaining the initial image of the endoscope and the three-axis angular velocity signal" specifically comprises: collecting a raw image stream as the initial image through the image sensor of the endoscope; collecting the three-axis angular velocity signal when the endoscope moves through the gyroscope integrated with the endoscope.

[0008] As an improvement of the embodiment of the present application, the "solving the attitude according to the three-axis angular velocity signal by using the quaternion differential equation to obtain the attitude quaternion, and updating the attitude quaternion in real time" specifically comprises: solving the attitude according to the quaternion differential equation to obtain the attitude quaternion, updating the attitude quaternion in real time, and performing normalization processing on the updated attitude quaternion within a preset period, wherein, is the differential of the attitude quaternion, q is the current attitude quaternion, is the quaternion multiplication operator, is the three-axis angular velocity vector; wherein w is the real part, x, y, z are imaginary parts of the attitude quaternion; wherein, is the three-axis angular velocity signal.

[0009] As an improvement of the embodiment of the present application, the "performing normalization processing on the updated attitude quaternion within a preset period" specifically comprises: calculating the module length of the updated attitude quaternion; and dividing each component of the attitude quaternion by the module length to obtain the normalized attitude quaternion.

[0010] As an improvement of the embodiment of the present application, the "interpolating the attitude quaternions of adjacent time points within a frame exposure time to obtain the line-level quaternion, and generating the line-level rotation matrix according to the line-level quaternion" specifically comprises: obtaining the center exposure time of each row of pixels in the initial image and the attitude sample of adjacent time points within a frame exposure time, obtaining the line interpolation coefficient according to the center exposure time of each row of pixels in the initial image and the attitude sample; then calculating the included angle of the adjacent attitude quaternions, and when the included angle is less than a preset included angle, obtaining the line-level quaternion according to linear interpolation and normalization, and when the included angle is greater than or equal to the preset included angle, obtaining the line-level quaternion according to spherical linear interpolation; converting the line-level quaternion into a three-dimensional rotation matrix, and then combining the intrinsic matrix of the image sensor in the endoscope to generate the line-level rotation matrix.

[0011] As an improvement of the embodiment of the present application, the "performing back-rotation and compensation on the initial image based on the line-level rotation matrix to obtain the back-rotated image" specifically comprises: The inverse mapping relationship is constructed based on the row-level rotation matrix, and floating point coordinates corresponding to the target pixels of the rectified image in the initial image are calculated; four integer grid points around the floating point coordinates are determined and offset is calculated, and the rectified image is obtained by linear interpolation in the horizontal direction and the vertical direction according to the gray values of the four grid points and the offset.

[0012] As an improvement of the embodiment of the present application, in the process of acquiring the initial image and the three-axis angular velocity signal, the time difference between the sampling time of the three-axis angular velocity signal and the row exposure center time of the initial image is calculated; and the sampling time of the three-axis angular velocity signal is offset compensated according to the time difference.

[0013] To achieve one of the above-mentioned purposes, an embodiment of the present application provides an image rectification device of an endoscope, comprising the following modules: A data acquisition module is configured to acquire an initial image of the endoscope and a three-axis angular velocity signal; A pose solving module is configured to solve the pose by using a quaternion differential equation according to the three-axis angular velocity signal to obtain a pose quaternion, and update the pose quaternion in real time; A rectification module is configured to interpolate the pose quaternions of adjacent time points to obtain row-level quaternions within one exposure time, generate a row-level rotation matrix according to the row-level quaternions, and rectify and compensate the initial image by using inverse mapping and bilinear interpolation based on the row-level rotation matrix to obtain a rectified image.

[0014] To achieve one of the above-mentioned purposes, an embodiment of the present application provides a storage medium storing program instructions, which, when executed, implement the image rectification method of the endoscope according to any one of the above.

[0015] To achieve one of the above-mentioned purposes, an embodiment of the present application provides an electronic terminal comprising a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the image rectification method of the endoscope according to any one of the above.

[0016] The image rectification method, device, storage medium and terminal of the endoscope provided by the embodiment of the present application have the following advantages: the present application uses quaternions to represent the spatial pose, effectively avoids the gimbal lock and the resulting functional failure in the traditional Euler angle pose solving, and continuously solves the pose in any angle range, so as to track the rapid flipping or rotating action of the endoscope; the quaternions can be efficiently updated by linear multiplication and addition, without multiple trigonometric function calculations, thereby improving the stability and real-time performance of the pose solving.

[0017] The application solves the posture by the gyro posture signal and carries out dynamic compensation by interpolation algorithm according to the solving result, effectively eliminates the jelly effect caused by the line-by-line exposure, avoids the delay, the insufficient real-time performance and the problem that it is difficult to integrate in the limited space existing in the traditional software algorithm or external micro control unit post-processing, and better meets the imaging needs of the medical endoscope clinical operation. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the image righting method of the endoscope according to the application; Figure 2 is a schematic diagram of the endoscope image before righting in the embodiment of the application; Figure 3 is a schematic diagram of the endoscope image after righting in the embodiment of the application; Figure 4 is a schematic diagram of the image righting device of the endoscope according to the application; Figure 5 is a schematic diagram of the structure of the electronic terminal according to the application. DETAILED DESCRIPTION

[0019] The application will be described in detail below with reference to the specific embodiments shown in the drawings. However, these embodiments do not limit the application, and the changes in structure, method or function made by those skilled in the art based on these embodiments are included in the protection scope of the application.

[0020] The scope of the embodiments herein includes the entire scope of the claims, and all available equivalents of the claims. Herein, the terms "first", "second" and the like are only used to distinguish one element from another element, and do not require or imply any actual relationship or order between the elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the structure, device or equipment comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such structure, device or equipment. Without more limitation, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the structure, device or equipment comprising the element. Various embodiments herein are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between various embodiments can be referred to each other.

[0021] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings and are used only for the convenience of describing this document and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements, or direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0022] This invention provides an image alignment method for an endoscope, such as... Figure 1 As shown, it includes the following steps: Step 101: Acquire the initial images and triaxial angular velocity signals from the endoscope; In practice, the image sensor of the endoscope can be used to acquire the raw image stream as the initial image. This image sensor needs to be matched with the endoscope's optical system to acquire scene information captured in real time by the endoscope's probe and output it as a raw image stream. A gyroscope integrated with the endoscope acquires the three-axis angular velocity signals during endoscope movement. This gyroscope needs to be pre-calibrated with the endoscope's probe to ensure its detection axis is consistent with the endoscope's motion axis, thus acquiring angular velocity data generated along three orthogonal directions when the endoscope undergoes flipping, rotation, or other movements during clinical operations. During the acquisition of the initial image and the three-axis angular velocity signals, the time difference between the sampling time of the three-axis angular velocity signals and the line exposure center time of the initial image is calculated based on this time difference. The sampling time of the three-axis angular velocity signals is then offset compensated based on this time difference. This avoids deviations in attitude calculation and image processing due to asynchronous data acquisition, ensuring that the initial image and the three-axis angular velocity signals reflect the endoscope's imaging and motion states at the same moment.

[0023] Step 102: Based on the triaxial angular velocity signals, attitude quaternions are obtained by performing attitude calculations using quaternion differential equations, and the attitude quaternions are updated in real time; specifically, this includes: based on the quaternion differential equations... Attitude calculation is performed to obtain attitude quaternions. These attitude quaternions are updated in real time, and normalized within a preset period. Let q be the differential of the attitude quaternion, and q be the current attitude quaternion. This is the quaternion multiplication operator. It is a three-axis angular velocity vector; , where w is the real part, and x, y, and z are the imaginary parts of the attitude quaternion, respectively; ,in, This is a three-axis angular velocity signal.

[0024] Here, the phrase "normalizing the updated attitude quaternions within a preset period" specifically includes: through... Calculate the modulus of the updated attitude quaternion; then divide each component of the attitude quaternion by the modulus to obtain the normalized attitude quaternion.

[0025] In practice, attitude calculation and updating can be implemented on a Field Programmable Gate Array (FPGA) hardware platform, fully utilizing its parallel processing capabilities and efficient logic operation characteristics to ensure real-time performance and stability. The FPGA establishes a data connection with the gyroscope through its integrated I2C or SPI communication module, receiving triaxial angular velocity signals in real time and temporarily storing the data in an internal FIFO buffer to avoid data loss or delay. Attitude calculation uses a hardware description language to embed quaternion operation logic within the FPGA. Based on the quaternion differential equation, it performs hardware-level quaternion multiplication operations with the current attitude quaternion and the triaxial angular velocity vector to complete the attitude quaternion differential calculation, thereby achieving real-time updates of the attitude quaternion. Simultaneously, the FPGA can perform normalization operations on the updated attitude quaternion at preset intervals. First, the magnitude of the attitude quaternion is calculated using hardware circuitry. Then, the ratio of each component to the magnitude is output through division logic to obtain the normalized attitude quaternion. The quaternion multiplication strictly follows Hamilton's multiplication rule. The timing synchronization between angular velocity vector processing and quaternion operations is controlled by the FPGA's internal clock signal. It can be understood that this hardware implementation method abandons the complex process in traditional software operations, replacing multiple trigonometric function calculations with linear multiplication and addition logic, which significantly improves the computational efficiency. Moreover, the entire process does not require an external processor, ensuring the continuity and accuracy of attitude calculation.

[0026] Step 103: interpolating the attitude quaternions of adjacent time points to obtain a row-level quaternion, generating a row-level rotation matrix according to the row-level quaternion, specifically including: obtaining the center exposure time of each row of pixels in the initial image and the attitude sample of adjacent time points within a frame exposure time, obtaining the row interpolation coefficient according to the center exposure time of each row of pixels in the initial image and the attitude sample; then calculating the included angle of adjacent attitude quaternions, when the included angle is less than a preset included angle, obtaining the row-level quaternion according to linear interpolation and normalization, when the included angle is greater than or equal to the preset included angle, obtaining the row-level quaternion according to spherical linear interpolation; converting the row-level quaternion into a three-dimensional rotation matrix, and then combining the intrinsic matrix of the image sensor in the endoscope to generate a row-level rotation matrix.

[0027] In practice, the attitude sample of adjacent time points can be 、 wherein, represents the i-th time point, represents the corresponding attitude quaternion at the time point . represents the i+1-th time point, represents the corresponding attitude quaternion at the time point . The center exposure time of the j-th row of pixels in the initial image is ; and the interpolation coefficient of the j-th row of pixels in the initial image is . The included angle of the adjacent attitude quaternions is , wherein, is the dot product. When the included angle is less than a preset included angle, the row-level quaternion is obtained according to linear interpolation and normalization , wherein norm( ) represents the normalization operation, is the conjugate of the attitude quaternion at the time point , the quaternion conjugate is the real part unchanged and the imaginary part reversed, which is used for spherical linear interpolation and other operations. When the included angle is greater than or equal to the preset included angle, the row-level quaternion is obtained according to spherical linear interpolation .

[0028] Then, the three-dimensional rotation matrix can be obtained through the standard right-handed system quaternion, and then the row-level rotation matrix is generated by combining the intrinsic matrix of the image sensor in the endoscope , wherein is the inverse matrix of the intrinsic matrix .

[0029] The inverse mapping and bilinear interpolation are used to rectify and compensate the initial image based on the row-level rotation matrix to obtain a rectified image, specifically comprising: constructing an inverse mapping relationship based on the row-level rotation matrix, and calculating a floating-point coordinate corresponding to a target pixel in the rectified image in the initial image; determining four integer grid points around the floating-point coordinate and calculating an offset, and performing linear interpolation in the horizontal direction and the vertical direction according to the gray values of the four grid points and the offset to obtain the rectified image.

[0030] In practice, the inverse mapping relationship , is a scaling factor, used to convert homogeneous coordinates into non-homogeneous coordinates; , v is a floating-point coordinate component in the initial image corresponding to the target pixel in the rectified image; , is a coordinate component of the target pixel in the rectified image; is the inverse matrix of the row-level rotation matrix , used to construct an inverse mapping relationship from the rectified image to the initial image, and realize reverse lookup of pixel coordinates; the homogeneous coordinates and respectively represent the homogeneous coordinate form of the pixels in the initial image and the rectified image, and through such matrix operation, the corresponding floating-point position of each pixel in the rectified image in the initial image can be determined, providing coordinate data for the bilinear interpolation.

[0031] Table 1 Four integer grid points around the floating-point coordinate and related information

[0032] The four integer grid points around the floating-point coordinate and their corresponding coordinates and gray values are shown in Table 1, and the offset , represents the offset in the horizontal direction, represents the offset in the vertical direction; , is the coordinate of the upper left integer grid point around the floating-point coordinate (u, v), , represents rounding down. Then, linear interpolation is performed in the horizontal direction and the vertical direction according to the gray values of the four grid points and the offset to obtain the rectified image, and the horizontal direction interpolation is specifically: ; ; and the vertical direction interpolation is specifically: , and after merging . Here, the intermediate results and are obtained by linearly interpolating the gray values of adjacent grid points in the horizontal direction according to the offset Then, based on the offset in the vertical direction right and Linear interpolation is used to obtain the grayscale values ​​of the pixels in the corrected image, thereby achieving image correction compensation.

[0033] Here, the initial image is as follows: Figure 2 As shown, the image exhibits viewpoint shift and geometric distortion due to changes in the spyscope's posture; the corrected image is as follows. Figure 3 As shown, after the image is corrected by the image correction method described in this invention, the geometric structure of the image is restored to normal and the viewing angle is more regular. It can be understood that this invention can effectively eliminate the image distortion caused by the movement of the endoscope through row-level pose compensation, so that the image remains geometrically consistent and the accuracy of image correction is guaranteed. It solves the problems of image distortion and viewing angle shift caused by dynamic changes in the endoscope pose. Moreover, row-level compensation can effectively eliminate the rolling shutter effect caused by progressive exposure.

[0034] By acquiring two sets of chessboard images with and without row-level compensation enabled, the geometric consistency of the grid was compared, meeting the requirements of reprojection error ≤ 0.5px, straightness change ≤ 0.3px, and angle deviation ≤ 0.2°. The consistency between attitude change and gyroscope measurements was verified by simultaneously acquiring gyroscope and image data through a low-speed rotating motor module, and the image center drift after compensation was verified to be ≤ 0.5px, with the row-level update frequency and image sensor error ≤ 5µs. The row number correspondence was recorded using an LED row scanning light source, verifying a row timing error ≤ 2µs. Simultaneously, static video was recorded using a fixed module, and brightness variance was calculated, verifying that attitude calculation jitter did not introduce image flicker, with an average brightness variance ≤ 1gray. It can be understood that the image straightening method described in this invention meets the qualified standards in terms of geometric stability, straightness, row timing synchronization, row attitude phase error, static brightness variance, and low-speed rotation drift. This avoids the problems of large delays, insufficient real-time performance, and difficulty in integration in confined spaces inherent in traditional software algorithms or external microcontroller post-processing, and better meets the imaging needs of clinical medical endoscopy operations.

[0035] The present invention also provides an image alignment device for an endoscope, such as... Figure 4 As shown, it includes the following modules: The data acquisition module 201 is used to acquire the initial image and triaxial angular velocity signal of the endoscope; The attitude calculation module 202 is used to calculate the attitude quaternion by using quaternion differential equations based on the triaxial angular velocity signal, and to update the attitude quaternion in real time. The alignment module 203 is used to interpolate the pose quaternions of adjacent time points within one frame of exposure time to obtain row-level quaternions, generate a row-level rotation matrix based on the row-level quaternions, and perform alignment and compensation on the initial image based on the row-level rotation matrix using inverse mapping and bilinear interpolation to obtain an aligned image.

[0036] The present invention also provides a storage medium storing program instructions that, when executed, implement the endoscope image alignment method as described in any of the preceding claims.

[0037] The present invention also provides an electronic terminal, such as Figure 5 As shown, it includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the image repositioning method of the endoscope as described in any of the preceding claims.

[0038] This invention can be an apparatus, method, and / or computer program product. A computer program product may include a readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0039] Storage media can be tangible devices that hold and store instructions for use by instruction execution devices. Storage media can include, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0040] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0041] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image alignment method for an endoscope, characterized in that, Includes the following steps: Acquire initial images and triaxial angular velocity signals from the endoscope; The attitude quaternion is obtained by using quaternion differential equations to solve the attitude based on the three-axis angular velocity signal, and the attitude quaternion is updated in real time. Within one frame of exposure time, the pose quaternions at adjacent time points are interpolated to obtain row-level quaternions. A row-level rotation matrix is ​​generated based on the row-level quaternions. Based on the row-level rotation matrix, the initial image is corrected and compensated using inverse mapping and bilinear interpolation to obtain a corrected image.

2. The endoscope image alignment method according to claim 1, characterized in that, The "acquiring of initial images and triaxial angular velocity signals of the endoscope" specifically includes: acquiring the original image stream as the initial image through the endoscope's image sensor; and acquiring the triaxial angular velocity signals of the endoscope during its movement through a gyroscope integrated with the endoscope.

3. The endoscope image alignment method according to claim 1, characterized in that, The phrase "obtaining attitude quaternions by performing attitude calculations using quaternion differential equations based on the triaxial angular velocity signals, and updating the attitude quaternions in real time" specifically includes: calculating attitude quaternions using quaternion differential equations... Attitude calculation is performed to obtain attitude quaternions. These attitude quaternions are updated in real time, and normalized within a preset period. Let q be the differential of the attitude quaternion, and q be the current attitude quaternion. This is the quaternion multiplication operator. It is a three-axis angular velocity vector; , where w is the real part, and x, y, and z are the imaginary parts of the attitude quaternions; ,in, This is a three-axis angular velocity signal.

4. The endoscope image alignment method according to claim 3, characterized in that, The phrase "normalizing the updated attitude quaternions within a preset period" specifically includes: through... Calculate the modulus of the updated attitude quaternion; then divide each component of the attitude quaternion by the modulus to obtain the normalized attitude quaternion.

5. The image alignment method for an endoscope according to claim 1, characterized in that, The phrase "interpolating the pose quaternions of adjacent time points within one frame of exposure time to obtain row-level quaternions, and generating a row-level rotation matrix based on the row-level quaternions" specifically includes: Within one frame of exposure time, acquire pose samples from adjacent moments and the center exposure time of each row of pixels in the initial image. Obtain row interpolation coefficients based on the pose samples and the center exposure time of each row of pixels in the initial image. Then, calculate the angle between adjacent pose quaternions. When the angle is less than a preset angle, obtain row-level quaternions based on linear interpolation and normalization. When the angle is greater than or equal to the preset angle, obtain row-level quaternions based on spherical linear interpolation. Convert the row-level quaternions into a three-dimensional rotation matrix, and then combine it with the intrinsic parameter matrix of the image sensor in the endoscope to generate a row-level rotation matrix.

6. The image alignment method for an endoscope according to claim 1, characterized in that, The phrase "using inverse mapping and bilinear interpolation to correct and compensate the initial image based on the row-level rotation matrix to obtain a corrected image" specifically includes: The inverse mapping relationship is constructed based on the row-level rotation matrix. The floating-point coordinates corresponding to the target pixel of the straightened image in the initial image are calculated. Four integer grid points around the floating-point coordinates are determined and the offset is calculated. The straightened image is obtained by linear interpolation in both the horizontal and vertical directions based on the gray values ​​of the four grid points and the offset.

7. The image alignment method for an endoscope according to claim 1, characterized in that, During the acquisition of the initial image and the three-axis angular velocity signal, the time difference between the sampling time of the three-axis angular velocity signal and the line exposure center time of the initial image is calculated based on the line exposure center time of the initial image; and the sampling time of the three-axis angular velocity signal is offset compensated according to the time difference.

8. An image alignment device for an endoscope, characterized in that, Includes the following modules: The data acquisition module is used to acquire the initial images and triaxial angular velocity signals of the endoscope; The attitude calculation module is used to calculate the attitude quaternion by using quaternion differential equations based on the three-axis angular velocity signals, and to update the attitude quaternion in real time. The alignment module is used to interpolate the pose quaternions of adjacent time points within one frame of exposure time to obtain row-level quaternions, and generate a row-level rotation matrix based on the row-level quaternions; based on the row-level rotation matrix, the initial image is aligned and compensated using inverse mapping and bilinear interpolation to obtain an aligned image.

9. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they implement the endoscope image alignment method as described in any one of claims 1 to 7.

10. An electronic terminal, characterized in that, It includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the image alignment method of the endoscope as described in any one of claims 1 to 7.

Images