FPGA-based endoscope image real-time distortion correction acceleration method

Abstract

The application discloses an endoscope image real-time distortion correction acceleration method based on FPGA, and the method comprises the following steps: image segmentation is performed on the endoscope image to determine a primary partition; the attention score of the primary partition is calculated to determine the attention level; the primary partition is iteratively segmented to determine one or more partition-level image partitions; the number of first pixel points in the n-level partition is determined according to the partition level n; the to-be-corrected pixel points of the n-level partition are determined according to the number of first pixel points, the distortion correction of the to-be-corrected pixel points is performed, and the pixel coordinates of the to-be-corrected pixel points after correction are determined; the remaining pixel points in the n-level partition are subjected to interpolation transformation according to the pixel coordinates of the to-be-corrected pixel points in the n-level partition before and after correction, and the display image of the n-level partition is determined; and the display images of all image partitions are spliced to generate an endoscope correction image. The image correction is performed through the distortion correction and interpolation transformation method, and the image distortion correction processing speed can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an FPGA-based method for accelerating real-time distortion correction of endoscopic images. Background Technology

[0002] Electronic endoscopes have become a core imaging device used in medical examinations such as breast and gastric examinations. To obtain a wide field of view, endoscopes generally use short focal length wide-angle lenses. However, the resulting barrel distortion causes distortion of the anatomical structures at the edges of the image. Clinical studies have shown that in a 5mm diameter laparoscope, the distortion error in the image area 60% of the distance from the center point can reach 8%-12%, seriously affecting the doctor's judgment of organ size and spatial position.

[0003] Existing image distortion correction methods (such as PC-based OpenCV algorithms) suffer from high processing latency (>100ms) and high power consumption, which cannot meet the real-time requirements of endoscopic video streams.

[0004] Therefore, the existing technology has defects and urgently needs improvement. Summary of the Invention

[0005] In view of the above problems, the purpose of this invention is to provide an FPGA-based method for accelerating real-time distortion correction of endoscopic images, which can improve the speed of image distortion correction processing while ensuring image display quality.

[0006] The first aspect of this invention provides an FPGA-based method for accelerating real-time distortion correction of endoscopic images, comprising:

[0007] Acquire endoscopic images;

[0008] The endoscopic image is segmented based on a preset partitioning rule to determine the first-level partition d. 1(i) ;

[0009] Calculate the first-level partition d 1(i) The attention score determines the first-level partition d. 1(i) Attention level;

[0010] Based on the level of attention, the first-level partition d is... 1(i) Perform iterative segmentation to determine one or more image partitions at different partition levels;

[0011] Determine the n-level partition d based on the partition level n. n(i) Number of first pixels M 1(n) ;

[0012] Based on the first pixel count M 1(n) Determine n-level partition dn(i) The pixels to be corrected;

[0013] The pixel to be corrected is subjected to distortion correction using a preset correction method, and the corrected pixel coordinates are determined.

[0014] Based on the pixel coordinates of the pixels to be corrected before and after correction in the n-level partition, the remaining pixels in the n-level partition are interpolated and transformed to determine the display image of the n-level partition;

[0015] The displayed images from all image partitions are stitched together to generate an endoscope correction image.

[0016] This plan also includes:

[0017] A three-dimensional image model is constructed based on the endoscopic image to determine the actual coordinates of each pixel in the endoscopic image.

[0018] A two-dimensional coordinate system is constructed based on the display interface to determine the display coordinates of each pixel in the endoscopic image;

[0019] Bind the actual coordinates and display coordinates of the same pixel;

[0020] The three-dimensional image model is identified by a preset endoscope recognition model, and the three-dimensional image model is divided into multiple recognition regions, and the region level of each recognition region is determined.

[0021] In this scheme, the calculation of the first-level partition d 1(i) The attention score determines the first-level partition d. 1(i) The levels of attention include:

[0022] The first-level partition d 1(i) The four vertices and the center point are determined as feature point j;

[0023] The region level Q of feature point j is determined based on the recognition region where the actual coordinates of feature point j are located. j ;

[0024] According to the region level Q of the feature point j j The first-level partition d is calculated according to the level. 1(i) Attention score P;

[0025] P = k a ×M Q(max) +Σ[k b(j) ×Q j ];

[0026] Among them, M Q(max) k represents the number of regions where the feature point falls within different levels of the region. a and kb(j) All are weighting coefficients, 1≤j≤5;

[0027] The attention level of the first-level partition is determined based on the attention level corresponding to the attention score P.

[0028] In this scheme, the first-level partition d is classified according to the level of attention. 1(i) Perform iterative segmentation, including:

[0029] When the attention level of the first-level partition is 1, no action is taken;

[0030] When the attention level of the first-level partition is greater than 1, the midpoint between adjacent vertices in the first-level partition is determined as an auxiliary point;

[0031] The primary partition is divided based on the auxiliary points to determine the secondary partition d. 2(i) .

[0032] This plan also includes:

[0033] Calculate the d of the n-level partition with partition level n. n(i) Attention score and attention level;

[0034] For n-level partitions d with a attention level greater than or equal to n+1 n(i) Perform partitioning to determine n+1 level partitions d n+1(i) ;

[0035] The iterative segmentation ends when there are no partitions of level n or higher than n+1, or when the preset number of segmentations is reached.

[0036] In this scheme, the step of determining the n-level partition d based on the partition level n is described. n(i) Number of first pixels M 1(n) ,include:

[0037] When n=1, M 1(n) =5;

[0038] When 1 < n ≤ n a At that time, M 1(n) = (2n-1) 2 ;

[0039] When n > n a At that time, M 1(n) =N (n) ;

[0040] Where, n a N is the first preset partition level. (n) This represents the total number of pixels in the n-level partition.

[0041] In this scheme, the step of determining the number of first pixels M...1(n) Determine n-level partition d n(i) The pixels to be corrected include:

[0042] Step 1, partition n-level partition d n(i) The feature points were identified as the pixels to be corrected.

[0043] Step 2, count the number M of pixels to be corrected. 2(n) ;

[0044] Step 3, when M 2(n) <M 1(n) When this is the case, the midpoint of adjacent feature points and / or auxiliary points is determined as the auxiliary point;

[0045] Step 4: Determine the auxiliary points as pixels to be corrected, and the number of pixels to be corrected is M. 2(n) Update;

[0046] Step 5: Repeat steps 3 and 4 until M. 2(n) =M 1(n) Determine the n-level partition d n(i) All pixels to be corrected.

[0047] This plan also includes:

[0048] When M 1(m) >k c *N (m) When the m-level partition is defined, the partition level corresponding to the m-level partition is determined as the first partition level m;

[0049] Among them, M 1(m) N represents the number of first pixels in the m-level partition. (m) k represents the total number of pixels in the m-level partition. c k is the influence coefficient. c ∈(0,1], m≤n;

[0050] The preset number of divisions is adjusted based on the first partition level m.

[0051] In this scheme, the step of performing interpolation transformation on the remaining pixels in the n-level partition based on the pixel coordinates of the pixels to be corrected before and after correction, and determining the display image of the n-level partition, includes:

[0052] When the partition level of the n-level partition is lower than the second preset partition level, bilinear interpolation is performed on the remaining pixels in the n-level partition based on the pixel coordinates of the pixels to be corrected before and after correction; otherwise, bicubic interpolation is performed on the remaining pixels in the n-level partition to determine the corrected pixel coordinates of the remaining pixels.

[0053] The display image of the n-level partition is determined based on the pixel coordinates of the pixels to be corrected and the remaining pixels after correction in the n-level partition.

[0054] This plan also includes:

[0055] Based on the partition level n, image partitions with partition sizes smaller than the corresponding preset partition sizes will be marked on the display interface, and the partition E to be reconstructed will be determined. a(n) ;

[0056] Generate a reorganization frame F based on the preset partition specifications corresponding to partition level n. (n) ;

[0057] Through the recombination frame F (n) Treating the reorganized partition E (n) Perform a traversal and calculate the recombination frame F. (n) The percentage of the partition to be reorganized at each traversal position;

[0058] After the traversal is complete, the maximum percentage of the partition to be reorganized is compared with the preset percentage threshold.

[0059] If the maximum proportion of the partition to be reorganized is greater than or equal to the preset proportion threshold, then the partition to be reorganized at the traversal position corresponding to the maximum value of the preset proportion threshold is reorganized.

[0060] The partition reorganization process filters the traversal positions corresponding to the partition to be reorganized, and selects the maximum proportion of the next partition to be reorganized area from the remaining traversal positions to verify the partition reorganization conditions.

[0061] If the maximum percentage of the partition to be reorganized is less than the corresponding preset percentage, then the reorganization of the partition to be reorganized at the current partition level will end, and the reorganization of the partition to be reorganized at the next partition level will begin.

[0062] This invention discloses an FPGA-based method for accelerating real-time distortion correction of endoscopic images. The method includes: segmenting the endoscopic image to determine primary partitions; calculating the attention score of each primary partition to determine the attention level; iteratively segmenting the primary partitions to determine one or more image partitions of different partition levels; determining the number of first pixels in each n-level partition based on the partition level n; determining the pixels to be corrected in each n-level partition based on the number of first pixels; performing distortion correction on the pixels to be corrected to determine the corrected pixel coordinates; interpolating the remaining pixels in each n-level partition based on the pixel coordinates of the pixels to be corrected before and after correction to determine the display image of the n-level partition; and stitching together the display images of all image partitions to generate a corrected endoscopic image. This invention uses distortion correction and interpolation methods for image correction, which can effectively improve the processing speed of image distortion correction. Attached Figure Description

[0063] Figure 1 A flowchart of the FPGA-based real-time distortion correction acceleration method for endoscopic images provided by the present invention is shown.

[0064] Figure 2 The flowchart of the method for determining the attention level based on the first-level partition provided by the present invention is shown;

[0065] Figure 3 The flowchart of the method for determining pixels to be corrected in an n-level partition provided by the present invention is shown. Detailed Implementation

[0066] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0067] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0068] Figure 1 A flowchart of the FPGA-based real-time distortion correction acceleration method for endoscopic images provided by the present invention is shown.

[0069] like Figure 1 As shown, this invention discloses an FPGA-based method for accelerating real-time distortion correction of endoscopic images, comprising:

[0070] S102, acquire endoscopic images;

[0071] S104, Perform image segmentation on the endoscopic image based on preset partitioning rules to determine the first-level partition d. 1(i) ;

[0072] S106, Calculate the first-level partition d 1(i) Attention score determines the first-level partition d 1(i) Attention level;

[0073] S108, Based on the level of attention, the first-level partition d 1(i) Perform iterative segmentation to determine one or more image partitions at different partition levels;

[0074] S110, determine the n-level partition d based on the partition level n. n(i) Number of first pixels M 1(n) ;

[0075] S112, based on the number of first pixels M 1(n) Determine n-level partition d n(i) The pixels to be corrected;

[0076] S114, perform distortion correction on the pixel to be corrected using a preset correction method, and determine the pixel coordinates of the pixel to be corrected after correction;

[0077] S116, Based on the pixel coordinates of the pixel to be corrected before and after correction in the n-level partition, perform interpolation transformation on the remaining pixels in the n-level partition to determine the display image of the n-level partition.

[0078] S118, stitches together the displayed images of all image partitions to generate an endoscope correction image.

[0079] According to embodiments of the present invention, a Field-Programmable Gate Array (FPGA) is a programmable logic chip that performs general-purpose functions, meaning it can be programmed to implement specific logic processing functions. FPGAs possess extremely high parallel processing capabilities, containing a large number of programmable logic units that can execute hundreds or even thousands of operations simultaneously, perfectly matching the pixel-level parallel computing requirements in image processing (such as filtering and convolution operations). Pixel-by-pixel processing can be achieved through FPGAs, with processing latency of only one or a few clock cycles, ensuring extremely low end-to-end latency (microsecond level) and meeting real-time requirements. With its hardware-level parallelism, ultra-low latency, reconfigurability, and high energy efficiency, FPGAs overcome the shortcomings of traditional CPU / GPU / ASIC solutions, making them a suitable hardware system for realizing functions such as real-time image enhancement and AI-assisted lesion detection.

[0080] The endoscopic equipment is used to inspect scenes including the breast, esophagus, and stomach. Endoscopic images are acquired through the camera module within the endoscope. Based on the inspection scene, a corresponding 3D image model is retrieved to generate a 3D image model of the current inspection scene. The recognition coordinates of each pixel in the endoscopic image are linked to their actual coordinates in the 3D image model. The system retrieves the preset partition size w×h for the first-level partition according to preset partitioning rules. The preset partition size is determined based on the partition level. The endoscopic image is then segmented into multiple first-level partitions of size w×h based on the preset partition size w×h. The four vertices (top left, top right, bottom left, and bottom right) and the center point within each first-level partition are identified as feature points. The region level of each feature point is determined by its actual coordinates. The attention score of the first-level partition is calculated to determine the corresponding attention level. When the attention level of a first-level partition is greater than 1, it is segmented into four second-level partitions of the same size. Subsequently, the attention level of the second-level partitions is calculated to determine whether further segmentation is necessary. By iteratively segmenting each level of partitions, for an n-level partition with partition level n, if its attention level is greater than or equal to n+1, the n-level partition is further segmented to determine the n+1-level partition. This iterative segmentation continues until no n-level partition with a partition level greater than or equal to n+1 exists, or the system's preset number of segmentations is reached. This process ends the iterative segmentation, determining one or more image partitions of different partition levels. Image partitions include level 1 to n-level partitions. When the attention level of a level 1 partition is 1, only level 1 partitions are included, and n ≥ 2.

[0081] The number of first pixels in the n-level partition is determined based on the partition level n. The feature points of the n-level partition are identified as pixels to be corrected. When the number of pixels to be corrected is less than the number of first pixels, the midpoint of adjacent feature points and / or known auxiliary points is identified as a new auxiliary point. The auxiliary points are used to supplement the pixels to be corrected.

[0082] Once the pixels to be corrected in all image partitions are determined, distortion correction is performed on the pixels in each partition using a system-preset distortion correction matrix. The corrected pixel coordinates are then used to transform the remaining pixels in the partition using interpolation transformations (such as bilinear or bicubic interpolation) based on these coordinates. This determines the corrected display image for each partition. Finally, the display images of each partition are stitched together using the vertex coordinates of each partition before correction to obtain the corrected endoscopic image. The system-preset distortion correction matrix is ​​trained using historically acquired endoscopic image samples.

[0083] This invention performs zonal correction on acquired endoscopic images. Based on the zonal level of each image zone, a corresponding number of pixels are selected for correction point-by-point. The remaining pixels within each image zone are then subjected to interpolation transformation. Compared to existing point-by-point correction methods for endoscopic images, this invention uses a relatively fast interpolation transformation method to correct only a portion of the pixels in the endoscopic image, reducing the number of pixels requiring point-by-point correction. While maintaining image display quality, it improves the speed of image distortion correction processing, meeting the real-time requirements of endoscopic image display.

[0084] According to an embodiment of the present invention, it further includes:

[0085] A three-dimensional image model is constructed based on the endoscopic image to determine the actual coordinates of each pixel in the endoscopic image.

[0086] Construct a two-dimensional coordinate system based on the display interface to determine the display coordinates of each pixel in the endoscopic image;

[0087] Bind the actual coordinates and display coordinates of the same pixel;

[0088] The three-dimensional image model is identified by a preset endoscope recognition model, and the three-dimensional image model is divided into multiple recognition regions, and the region level of each recognition region is determined.

[0089] It should be noted that the system collects image and video data generated during endoscopy use, and constructs a 3D image model of the detection scene (such as the breast, esophagus, and stomach) by combining the endoscopy's movement parameters. Using the initial detection point as the starting point of the 3D coordinate system, and combining it with the currently acquired endoscopic image, it determines the actual coordinates (including x, y, and z-axis coordinates) of each pixel in the endoscopic image. Simultaneously, it determines the center point of the display interface as the starting point of the 2D coordinate system, constructing a 2D coordinate system to determine the display coordinates (including x and y-axis coordinates) of each pixel in the endoscopic image on the display interface, and binds the display coordinates of the same pixel to its actual coordinates.

[0090] The preset endoscopic recognition model is trained using sample data from detection scenarios collected through networks and other means. It constructs 3D image models of detection scenarios such as the breast, esophagus, and stomach. Combining this with diagnostic information from the sample data, the probability of a lesion at each location in the 3D image model is determined. Based on the probability of a lesion at each location, the 3D image model is divided into recognition regions, and the region level of each recognition region is determined. The 3D image model is then input into the preset endoscopic recognition model. By comparing it with the corresponding sample 3D images, the recognition regions of the 3D image model and the region level of each recognition region are determined based on the distribution of recognition regions in the corresponding sample 3D images.

[0091] Figure 2 The flowchart of the method for determining the attention level based on the first-level partition provided by the present invention is shown.

[0092] like Figure 2 As shown, according to an embodiment of the present invention, the first-level partition d is calculated. 1(i) Attention score determines the first-level partition d 1(i) The levels of attention include:

[0093] S202, partition d of the first-level partition 1(i) The four vertices and the center point are determined as feature point j;

[0094] S204, Determine the region level Q of feature point j based on the actual coordinates of feature point j within the recognition region. j ;

[0095] S206, based on the region level Q of feature point j j Level calculation of first-level partition d 1(i) Attention score P;

[0096] P = k a ×M Q(max) +Σ[k b(j) ×Q j ];

[0097] Among them, M Q(max) k represents the number of regions where the feature point falls within different levels of the region. a and k b(j) All are weighting coefficients, 1≤j≤5;

[0098] S208. The attention level of the first-level partition is determined based on the attention level corresponding to the attention score P.

[0099] It should be noted that the first-level partition is a rectangular region, with its four vertices being the top-left, top-right, bottom-left, and bottom-right pixels within the rectangular region. The actual coordinates of feature point j determine which recognition region within the corresponding 3D image model the feature point is located in, and the region level of that recognition region is defined as the region level of the feature point.

[0100] The attention score ranges from 0 to 100, and there are 5 levels of attention. The attention score ranges corresponding to levels 1-5 are [0-60], (60-70], (70-80], (80-90] and (90-100) respectively.

[0101] Wherein, the weighting coefficient k a and k b(j) The specific value of k is set by those skilled in the art according to actual needs. b(j) For the regional level Q jThe weighting coefficients are different for each region level.

[0102] According to an embodiment of the present invention, the first-level partition d is classified according to the level of attention. 1(i) Perform iterative segmentation, including:

[0103] When the attention level of the first-level partition is 1, no action is taken;

[0104] When the attention level of a first-level partition is greater than 1, the midpoint between adjacent vertices in the first-level partition is determined as an auxiliary point.

[0105] The primary partition is divided based on auxiliary points to determine the secondary partition d. 2(i) .

[0106] It should be noted that there are 4 auxiliary points in the first-level partition. Connecting the auxiliary points in pairs will result in a set of vertical dividing lines. These dividing lines will then divide the first-level partition into four areas of the same size, resulting in four second-level partitions.

[0107] According to an embodiment of the present invention, it further includes:

[0108] Calculate the d of the n-level partition with partition level n. n(i) Attention score and attention level;

[0109] For n-level partitions d with a attention level greater than or equal to n+1 n(i) Perform partitioning to determine n+1 level partitions d n+1(i) ;

[0110] The iterative segmentation ends when there are no partitions of level n or higher than n+1, or when the preset number of segmentations is reached.

[0111] It should be noted that the partition level is determined by the number of partitions. That is, the partition level of the first-level partition obtained by the first partition is 1, the partition level of the second-level partition obtained by the second partition is 2, and the partition level of the nth partition obtained by the nth partition is n.

[0112] After determining the n-level partitions, the display coordinates of the feature points (four vertices and the center point) of each n-level partition are first determined. After converting the display coordinates of the feature points to their actual coordinates, the region level corresponding to each feature point is determined. Then, the attention score of the n-level partition is calculated based on the region level corresponding to each feature point, determining the attention level. Based on the attention level, it is determined whether to segment the n-level partition. Image partitions of each partition level are segmented iteratively until the iteration segmentation termination condition is met (i.e., there are no n-level partitions with a partition level greater than or equal to n+1, or the preset number of segmentations is reached), thereby determining image partitions of multiple partition levels. The preset number of segmentations is set by those skilled in the art according to actual needs.

[0113] According to an embodiment of the present invention, the n-level partition d is determined based on the partition level n. n(i) Number of first pixels M 1(n) ,include:

[0114] When n=1, M 1(n) =5;

[0115] When 1 < n ≤ n a At that time, M 1(n) = (2n-1) 2 ;

[0116] When n > n a At that time, M 1(n) =N (n) ;

[0117] Where, n a N is the first preset partition level. (n) This represents the total number of pixels in the n-level partition.

[0118] It should be noted that the number of first pixels differs in image partitions of different partition levels (including first-level partitions, second-level partitions, and n-level partitions, etc.). The partition level n corresponds to the n-level partition d. n(i) Number of first pixels M 1(n) .

[0119] Wherein, the first preset partition level n a The settings shall be configured by those skilled in the art according to actual needs, when n > n a When the accuracy requirement for the n-level partition is high, all pixels are identified as pixels to be corrected.

[0120] Figure 3 The flowchart of the method for determining pixels to be corrected in an n-level partition provided by the present invention is shown.

[0121] like Figure 3 As shown, according to an embodiment of the present invention, based on the number of first pixels M 1(n) Determine n-level partition d n(i) The pixels to be corrected include:

[0122] S302, partition n-level d n(i) The feature points were identified as the pixels to be corrected.

[0123] S304, Count the number of pixels to be corrected (M) 2(n) ;

[0124] S306, when M 2(n) <M 1(n)When this is the case, the midpoint of adjacent feature points and / or auxiliary points is determined as the auxiliary point;

[0125] S308, the auxiliary points are determined as the pixels to be calibrated, and the number of pixels to be calibrated is M. 2(n) Update;

[0126] S310, repeat S306 and S308 until M. 2(n) =M 1(n) Determine the n-level partition d n(i) All pixels to be corrected.

[0127] It should be noted that feature points (including the four vertices at the top left, top right, bottom left, and bottom right, as well as the center point) in the n-level partition are preferentially identified as pixels to be corrected. When the number of pixels to be corrected is M... 2(n) Less than the number of corresponding first pixels M 1(n) At this time, auxiliary points are determined to supplement the pixels to be corrected. Through one or more rounds of auxiliary point selection, until M is reached. 2(n) =M 1(n) Determine the n-level partition d n(i) All pixels to be corrected.

[0128] In each round of auxiliary point selection, adjacent feature points and / or auxiliary points have the same pixel distance in the same direction.

[0129] According to an embodiment of the present invention, it further includes:

[0130] When M 1(m) >k c *N (m) When the m-level partition is defined, the partition level corresponding to the m-level partition is determined as the first partition level m;

[0131] Among them, M 1(m) N represents the number of first pixels in the m-level partition. (m) k represents the total number of pixels in the m-level partition. c k is the influence coefficient. c ∈(0,1], m≤n;

[0132] The preset number of divisions is adjusted based on the first partition level m.

[0133] It should be noted that the influence coefficient k c The system settings are configured, and the specific value is adjusted according to the endoscope's state. When the endoscope is in motion, k... c The value of k is adjusted according to the movement speed; the faster the movement speed, the higher the value of k. c The smaller the value of k, the fewer the number of segmentations required to ensure the image can be updated in real time according to the movement of the endoscope; when the endoscope is stationary, k can be...c The value of is adjusted to the maximum value of 1 to ensure the display effect of the screen.

[0134] According to an embodiment of the present invention, the remaining pixels in the n-level partition are interpolated and transformed based on the pixel coordinates of the pixels to be corrected before and after correction in the n-level partition to determine the display image of the n-level partition, including:

[0135] When the partition level of the n-level partition is lower than the second preset partition level, bilinear interpolation is performed on the remaining pixels in the n-level partition based on the pixel coordinates of the pixels to be corrected before and after correction; otherwise, bicubic interpolation is performed on the remaining pixels in the n-level partition to determine the corrected pixel coordinates of the remaining pixels.

[0136] The display image of the n-level partition is determined based on the pixel coordinates of the pixels to be corrected and the remaining pixels after correction in the n-level partition.

[0137] It should be noted that when the partition level of the n-th level is less than the second preset partition level, it indicates that the attention level of this n-th level partition is low, and its impact on the display effect of the endoscopic image is small. In this case, a bilinear interpolation method with low computational cost but moderate accuracy is used to determine the corrected pixel coordinates of the remaining pixels within this n-th level partition. Conversely, when the partition level of the n-th level is greater than or equal to the second preset partition level, it indicates that the attention level of this n-th level partition is high, and its impact on the display effect of the endoscopic image is significant. In this case, a bicubic interpolation method with higher computational accuracy but higher computational cost is used to correct the pixel coordinates of the remaining pixels within this n-th level partition, thereby generating a smoother and more accurate display image.

[0138] The second preset partition level is set by those skilled in the art according to actual needs, and the second preset partition level is lower than the first preset partition level.

[0139] According to an embodiment of the present invention, it further includes:

[0140] Based on the partition level n, image partitions with partition sizes smaller than the corresponding preset partition sizes will be marked on the display interface, and the partition E to be reconstructed will be determined. a(n) ;

[0141] Generate a reorganization frame F based on the preset partition specifications corresponding to partition level n. (n) ;

[0142] By recombining frame F (n) Treating the reorganized partition E (n) Perform a traversal and calculate the recombined box F. (n) The percentage of the partition to be reorganized at each traversal position;

[0143] After the traversal is complete, the maximum percentage of the partition to be reorganized is compared with the preset percentage threshold.

[0144] If the maximum proportion of the partition to be reorganized is greater than or equal to the preset proportion threshold, then the partition to be reorganized at the traversal position corresponding to the maximum proportion threshold of the preset proportion will be reorganized.

[0145] The partition reorganization process filters the traversal positions corresponding to the partition to be reorganized, and selects the maximum proportion of the next partition to be reorganized area from the remaining traversal positions to verify the partition reorganization conditions.

[0146] If the maximum percentage of the partition to be reorganized is less than the corresponding preset percentage, then the reorganization of the partition to be reorganized at the current partition level will end, and the reorganization of the partition to be reorganized at the next partition level will begin.

[0147] It should be noted that the preset partition size is determined according to the partition level. For example, the preset partition size corresponding to the first-level partition is w×h, and the preset partition size corresponding to the second-level partition is 1 / 4w×1 / 4h, etc.

[0148] Image partitions in the display interface are grouped according to their partition levels, with priority given to analyzing the highest-level partitions. The partition specifications of each image partition are compared with their corresponding preset partition specifications. Partitions with partition specifications smaller than the corresponding preset specifications are identified as partitions to be reconstructed. A reconstructing frame is used to traverse the partitions to be reconstructed, calculating the ratio of the area occupied by all partitions to the frame size at each traversal position, determining the proportion of the partition area to be reconstructed at each traversal position. The maximum proportion of the partition area to be reconstructed is selected from all traversal positions for partition reconstructing condition verification. When the maximum proportion is greater than or equal to a preset area proportion threshold, the partition reconstructing condition is met, and partition reconstructing is performed on the corresponding traversal position. Traversal positions occupied during the reconstructing process are filtered out. The next maximum proportion of the partition area to be reconstructed is selected from the remaining traversal positions for partition reconstructing condition verification. This process continues until all traversal positions have been verified or the maximum proportion of the partition area to be reconstructed is less than the corresponding preset area proportion. At this point, the reconstructing of the partitions to be reconstructed at the current partition level ends, and the partitions not yet reconstructed are identified as reconstructed image partitions. The next partition level partition to be reconstructed is reconstructed. During the distortion correction of the pixels to be corrected and the interpolation transformation of other pixels, the reconstructed image partition is treated as a whole for image processing. During image processing, distortion correction is performed only on the pixels to be corrected within the reconstructed image partition, and interpolation transformation is performed only on other pixels within the reconstructed image partition.

[0149] The preset area percentage value is set by those skilled in the art according to actual needs.

[0150] All information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the "endoscopic images" involved in this disclosure were obtained under full authorization.

[0151] This invention discloses an FPGA-based method for accelerating real-time distortion correction of endoscopic images. The method includes: segmenting the endoscopic image to determine primary partitions; calculating the attention score of each primary partition to determine the attention level; iteratively segmenting the primary partitions to determine one or more image partitions of different partition levels; determining the number of first pixels in each n-level partition based on the partition level n; determining the pixels to be corrected in each n-level partition based on the number of first pixels; performing distortion correction on the pixels to be corrected to determine the corrected pixel coordinates; interpolating the remaining pixels in each n-level partition based on the pixel coordinates of the pixels to be corrected before and after correction to determine the display image of the n-level partition; and stitching together the display images of all image partitions to generate a corrected endoscopic image. This invention uses distortion correction and interpolation methods for image correction, which can effectively improve the processing speed of image distortion correction.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0153] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0155] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. An FPGA-based endoscope image real-time distortion correction acceleration method, characterized in that, The method comprises the following steps: acquiring an endoscope image; perform image segmentation on the endoscope image based on a preset partition rule to determine a first-level partition d 1(i) ; calculating a score of attention for the primary partition d 1(i) determining a level of attention for the primary partition d 1(i) ​ determining one or more image partitions of a second level of partitions based on the attention level of the first level of partitions 1(i) performing an iterative partitioning to determine one or more image partitions of a second level of partitions; determine a level-n partition d according to a partition level n n(i) M is the number of first pixel points 1(n) ; According to the first pixel point quantity M 1(n) The n-level partition d n(i) The pixel points to be corrected; correcting the to-be-corrected pixel points through a preset correction method to determine the pixel coordinates of the to-be-corrected pixel points after correction; performing interpolation transformation on the remaining pixel points in the n-level partition according to the pixel coordinates of the to-be-corrected pixel points before and after correction to determine the display image of the n-level partition; splicing the display images of all the image partitions to generate an endoscope correction image; The method further comprises the following steps: constructing a three-dimensional image model according to the endoscope image to determine the actual coordinates of each pixel point in the endoscope image; constructing a two-dimensional coordinate system according to the display interface to determine the display coordinates of each pixel point in the endoscope image; binding the actual coordinates and the display coordinates of the same pixel points; identifying the three-dimensional image model through a preset endoscope identification model, dividing the three-dimensional image model into multiple identification regions, and determining the region level of each identification region; calculating the attention score of the primary partition d 1(i) determining the attention level of the primary partition d 1(i) includes: determine the 4 vertices and the center point of the primary partition d 1(i) as feature points j; determining a region level Q of the feature point j according to a region in which an actual coordinate of the feature point j is located j ; According to the region level Q of the feature point j j The level calculates the first-level partition d 1(i) The attention score P of the feature point j ； Wherein, M Q(max) is the number of regions in which the feature points are in different levels of regions, k a and k b(j) are weight coefficients, 1≤j≤5; determining the attention level of the first-level partition according to the attention level corresponding to the attention score P; The first-level partition d is divided according to the attention level 1(i) The iterative segmentation is performed, comprising: when the attention level of the first-level partition is 1, no processing is performed; when the attention level of the first-level partition is greater than 1, the midpoint between adjacent vertices in the first-level partition is determined as an auxiliary point; According to the auxiliary point, the primary partition is segmented to determine a secondary partition d 2(i) ; Computing an n-level partition d of partition level n n(i) a score of attention and a level of attention determination of n+1 level partitions d n(i) performing the splitting, determining n+1 level partitions d n+1(i) ; when there is no n-level partition with a partition level greater than or equal to n+1 or the preset segmentation times are reached, the iteration segmentation is ended. 2.The FPGA-based endoscope image real-time distortion correction acceleration method of claim 1, wherein, The n-level partition d is determined according to the partition level n n(i) The first pixel point quantity M 1(n) , comprising: M = 5 when n = 1 1(n) =5; when 1 < n < n a M 1(n) = (2n - 1) 2 ; When n > n a M 1(n) = N (n) ; wherein n a is a first preset partition level, N (n) is a total number of pixel points in the n-level partition. 3.The FPGA-based endoscope image real-time distortion correction acceleration method of claim 1, wherein, The first pixel point number M 1(n) determining n-level partitions d n(i) the pixel points to be corrected, comprising: Step 1, the n-level partition d n(i) The feature points of the n-level partition d are determined as the pixel points to be corrected. Step 2, count the number of pixels M to be corrected 2(n) ; Step 3, when M 2(n) When M 1(n) the midpoint of the adjacent feature points and / or auxiliary points is determined as an auxiliary point; Step 4, determining the auxiliary points as the pixel points to be corrected, the number M of the pixel points to be corrected 2(n) updating; Step 5, repeat Step 3 and Step 4 until M 2(n) =M 1(n) , determine all the pixel points to be corrected in the n-th partition d n(i) . 4.The FPGA-based endoscope image real-time distortion correction acceleration method of claim 1, wherein, The method further comprises the following steps: When M 1(m) > k c * N (m) , the partition level corresponding to the m-th partition is determined as the first partition level m; wherein M 1(m) is the number of first pixel points in the m-level partition, N (m) is the total number of pixel points in the m-level partition, k c is the influence coefficient, k c ∈(0, 1], m ≤ n; adjusting the preset segmentation times according to the first partition level m. 5.The FPGA-based endoscopic image real-time distortion correction acceleration method of claim 1, wherein, The interpolation transformation on the remaining pixel points in the n-level partition according to the pixel coordinates of the to-be-corrected pixel points before and after correction to determine the display image of the n-level partition comprises the following steps: when the partition level of the n-level partition is less than a second preset partition level, performing bilinear interpolation on the remaining pixel points in the n-level partition according to the pixel coordinates of the to-be-corrected pixel points before and after correction; otherwise, performing bicubic interpolation on the remaining pixel points in the n-level partition to determine the pixel coordinates of the remaining pixel points after correction; determining the display image of the n-level partition according to the pixel coordinates of the to-be-corrected pixel points and the remaining pixel points after correction. 6.The FPGA-based endoscopic image real-time distortion correction acceleration method of claim 1, wherein, The method further comprises the following steps: According to the partition level n, the image partition whose partition specification in the display interface is less than the corresponding preset partition specification is marked, and a to-be-recombined partition E is determined a(n) ; The reorganization frame F is generated according to a preset partition specification corresponding to the partition level n (n) ; By the recombination frame F (n) The recombination partition E to be (n) Traverse, calculate the recombination frame F (n) The recombination partition area ratio of each traversal position after the traversal is completed, comparing the maximum area ratio of the to-be-reorganized partition region with a preset area ratio threshold value; if the maximum area ratio of the to-be-reorganized partition region is greater than or equal to the preset area ratio threshold value, reorganizing the to-be-reorganized partition corresponding to the traversal position of the maximum area ratio of the preset area ratio threshold value; filtering the traversal position corresponding to the to-be-reorganized partition occupied in the partition reorganization process, and selecting the maximum area ratio of the next to-be-reorganized partition region from the remaining traversal positions for partition reorganization condition verification; if the maximum area ratio of the to-be-reorganized partition is less than the corresponding preset area ratio value, ending the reorganization of the to-be-reorganized partition of the current partition level and reorganizing the to-be-reorganized partition of the next partition level.

Images