Image optimization method for confocal endoscope, and related device

Abstract

The present application relates to the field of image processing, and discloses an image optimization method for a confocal endoscope, and a related device. The method comprises: obtaining a data set to be calculated, wherein said data set is obtained by performing a data denoising operation on data to be processed; obtaining an alignment parameter on the basis of said data set; obtaining a correction target value on the basis of said data set; performing an alignment operation on a sine sampling data point set on the basis of the alignment parameter to obtain an aligned sine sampling data point set; and performing a correction operation on the aligned sine sampling data point set on the basis of the correction target value to obtain an optimization data set.

Description

A confocal endoscope image optimization method and related equipment

[0001] This application claims priority to the Chinese patent application filed by the applicant on November 28, 2023 with the Chinese Patent Office application number 202311601542.X and the invention name “A confocal endoscope image optimization method and related equipment”, the entire contents of which are incorporated by reference into this application, and all related applications cited and included in this application are an integral part of this application. Technical Field

[0002] This specification relates to the field of image processing, and more specifically, the present application relates to a confocal endoscope image optimization method and related equipment. Background Art

[0003] A confocal endoscope is a medical device that can be inserted into the human body through channels such as gastroscopes and colonoscopes to obtain local histological images for accurate diagnosis of small lesions, gastrointestinal lesions, and early gastrointestinal cancers. The scanning control module of a confocal endoscope has two key components: a resonant mirror and a galvanometer galvanometer. The resonant mirror is used to rapidly scan light horizontally (hence also called an X-galvanometer), while the galvanometer galvanometer is used to scan light vertically (hence also called a Y-galvanometer). The two work together to produce a two-dimensional image.

[0004] The resonant mirror operates by rotating back and forth along its axis through a fixed angle, turning in the opposite direction at the edge of the scanning range. The scanning directions of two adjacent rows are opposite, and due to the high scanning speed, it is difficult for the scanning starting points of the two adjacent rows to remain consistent, resulting in displacement between the two adjacent rows. The angular velocity of the sinusoidal characteristic during scanning is slow at the edges and fast in the center, resulting in overall distortion with stretching at the ends and compression in the middle. Related technologies lack an accurate image optimization method.

[0005] Summary of the Invention

[0006] The Summary of the Application introduces a series of simplified concepts that will be further described in the Detailed Description of the Invention. The Summary of the Application is not intended to limit the key features and essential features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0007] In a first aspect, the present application proposes a confocal endoscope image optimization method, the method comprising:

[0008] Obtaining a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed;

[0009] Obtain alignment parameters based on the above-mentioned data set to be calculated;

[0010] Obtaining a correction target value based on the above-mentioned data set to be calculated;

[0011] Performing an alignment operation on the sinusoidal sampling data point set based on the alignment parameters to obtain an aligned sinusoidal sampling data point set;

[0012] A correction operation is performed on the aligned sinusoidal sampling data point set based on the correction target value to obtain an optimized data set.

[0013] Optionally, the above method further includes:

[0014] Selecting a first data set and two second data sets from the calculation data set, wherein each data in the second data set is adjacent to a data in the first data set in the calculation data set, and second data in different second data sets and in the same order as the first data set are located on both sides of the first data in the first data set in the calculation data set;

[0015] Performing an offset operation on the second data set based on the first data set, and calculating an alignment cost;

[0016] The target alignment parameters are determined based on the alignment cost values.

[0017] Optionally, the selecting of a first data set and two second data sets from the calculation data sets includes:

[0018] Determine the starting row number and the ending row number of the target area in the image in the above calculation data set;

[0019] Perform segmentation based on the starting row number, the ending row number, and a preset number of segments to obtain the nearest odd or even row at the segment to form the first data set;

[0020] Second data adjacent to the first data in the first data set is acquired based on the first data set to form two second data sets.

[0021] Optionally, determining the starting row number and the ending row number of the target area in the image in the calculation data set includes:

[0022] Obtaining a pixel value histogram of the image data in the above-mentioned calculation data set;

[0023] Based on the above pixel value histogram, the OTSU algorithm is used to obtain the threshold information;

[0024] The starting row number and the ending row number are determined based on the threshold information.

[0025] Optionally, the above method further includes:

[0026] The alignment cost is calculated using the following formula:

[0027] Among them, P represents the number of lines used in the alignment operation, eid and sid represent the end line number and the start line number respectively, od p (id) represents the idth data point in the even row p under offset D, ed 1 p (id) and ed 2 p (id) represents the corresponding point of the idth data point in the odd row p in two different even row data sets, is a double summation symbol, indicating that all selected odd rows and each data point in these rows are to be iterated. and is the absolute value difference, representing the difference between the odd and even rows of a given data point under D offset, is the normalization factor.

[0028] Optionally, the above method further includes:

[0029] Obtain a uniform scanning sampling point data set and a sinusoidal sampling data point set;

[0030] Calculate the center difference information of the uniformly scanned sampling point data set and the sinusoidal sampling data point set;

[0031] Calculate the reference variable based on the above center value difference information, stretch coefficient, galvanometer scanning frequency and sine sampling frequency;

[0032] Determining a correction reference value based on the reference variable and the reference variable threshold information;

[0033] An approximate rounding operation is performed based on the above correction reference value and the sum of the central column information of the image in the sinusoidal sampling data point set to obtain a correction target value.

[0034] Optionally, the reference variable threshold information includes first threshold information and second threshold information, the second threshold information is greater than the first threshold information, and the correction reference value includes a first correction reference value, a second correction reference value, and a third correction reference value.

[0035] The above-mentioned determination of the correction reference value based on the above-mentioned reference variable and the above-mentioned reference variable threshold information includes:

[0036] When the reference variable is less than the first threshold value information, the first correction reference value is determined as the correction reference value; or, when the reference variable is greater than the second threshold value information, the second correction reference value is determined as the correction reference value:

[0037] The first correction reference value dn is determined based on the following formula 11 :

[0038] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency;

[0039] The second correction value dn is determined based on the following formula 12 :

[0040] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency;

[0041] The third correction value dn is determined based on the following formula: 13 :

[0042] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sine sampling frequency, α is the stretching coefficient, dn2 is the center difference information, and arcsin() is the inverse sine function.

[0043] Optionally, the above method further includes:

[0044] Performing a binarization operation on the image data to be calculated in the above-mentioned data set to be calculated to obtain a binary image set;

[0045] performing a morphological closing operation on the binary image data in the binary image set to obtain a closed image set;

[0046] Acquire height information, width information, and edge column information of the elliptical region in the closed image in the closed image set;

[0047] Correction parameters are calculated based on the height information, width information, and outermost column information, wherein the correction parameters include stretch coefficient information and center column information.

[0048] Optionally, the above method further includes:

[0049] Calculate the stretch coefficient information according to the height information and the width information;

[0050] The above center column information C is calculated according to the following formula:

[0051] Where C is the coordinate of the geometric center point of the image, c0 is the edge column information, H1 is the height information, W1 is the width information, and f sample is the sampling frequency, f scanis the scanning frequency, and arcsin() is the inverse sine function.

[0052] In a second aspect, the present application further proposes a confocal endoscope image optimization device, comprising:

[0053] A first acquisition unit is configured to acquire a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed;

[0054] A second acquisition unit, configured to acquire alignment parameters based on the dataset to be calculated;

[0055] A third acquisition unit is used to acquire a correction target value based on the data set to be calculated;

[0056] a fourth acquiring unit, configured to perform an alignment operation on the sinusoidal sampling data point set based on the alignment parameters to acquire an aligned sinusoidal sampling data point set;

[0057] A fifth acquisition unit is configured to perform a correction operation on the aligned sinusoidal sampling data point set based on the correction target value to obtain an optimized data set.

[0058] In summary, the confocal endoscope image optimization method of the embodiment of the present application includes: obtaining a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed; obtaining alignment parameters based on the data set to be calculated; obtaining a correction target value based on the data set to be calculated; performing an alignment operation on the sinusoidal sampling data point set based on the alignment parameters to obtain an aligned sinusoidal sampling data point set; performing a correction operation on the aligned sinusoidal sampling data point set based on the correction target value to obtain an optimized data set. The embodiment of the present application provides a confocal endoscope image optimization method, which calculates alignment parameters and correction target values ​​in a preprocessing mode, performs an alignment operation on the image data based on the alignment parameters, and performs a correction operation based on the correction target value. By switching between preprocessing and normal working modes, the endoscope can adapt to different usage scenarios, ensuring that high-quality image data can be obtained under any conditions. Through this method, the raw data collected by the confocal endoscope can be aligned and corrected to eliminate the problems of data misalignment and distortion caused by the resonant mirror scanning process.

[0059] The confocal endoscopic image optimization method proposed in this application, and other advantages, objectives and features of this application will be reflected in part through the following description, and in part will be understood by technical personnel in this field through research and practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present description. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0061] FIG1 is a schematic flow chart of a confocal endoscope image optimization method provided in an embodiment of the present application;

[0062] FIG2 is a schematic diagram of the spatial position of a resonant mirror during scanning according to an embodiment of the present application;

[0063] FIG3 is a schematic diagram of a reciprocating scanning sampling principle of a resonant mirror provided in an embodiment of the present application;

[0064] FIG4 is a schematic diagram showing the relationship between the angular velocity and the spatial position of a resonant mirror during scanning according to an embodiment of the present application;

[0065] FIG5 is a schematic diagram showing a principle for determining the starting and ending rows of an image valid area according to an embodiment of the present application;

[0066] FIG6 is a schematic diagram of a pixel histogram provided in an embodiment of the present application;

[0067] FIG7 is a schematic diagram of an alignment scenario with different offset values ​​provided by an embodiment of the present application;

[0068] FIG8 is a schematic diagram of an image to be calculated provided by an embodiment of the present application;

[0069] FIG9 is a schematic diagram of a binarized image provided in an embodiment of the present application;

[0070] FIG10 is a schematic diagram of a closed image provided by an embodiment of the present application;

[0071] FIG11 is a schematic diagram showing a principle of a correction algorithm provided in an embodiment of the present application;

[0072] FIG12 is a schematic diagram of an uncorrected image provided by an embodiment of the present application;

[0073] FIG13 is a schematic diagram of a corrected image provided by an embodiment of the present application;

[0074] Figure 14 is a structural schematic diagram of a confocal endoscope image optimization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0076] Figure 2 is a schematic diagram of the spatial position of a resonant mirror during scanning in the related art; Figure 3 is a schematic diagram of the principle of reciprocating scanning sampling by a resonant mirror in the related art; and Figure 4 is a schematic diagram of the relationship between angular velocity and spatial position during resonant mirror scanning in the related art. As shown in Figures 2 and 4, the resonant mirror operates by rotating back and forth along its axis through a certain angle, turning and rotating in the opposite direction when it reaches the edge of the scanning range. Furthermore, the angular velocity during scanning varies sinusoidally with spatial position.

[0077] During confocal endoscope scanning, sampling is generally performed at equal time intervals. Due to the reciprocating and sinusoidal characteristics of the resonant mirror during scanning, the original image obtained by sampling has the following problems: (1) the scanning directions of two adjacent rows are opposite to each other; (2) due to the high scanning speed, it is difficult to keep the scanning starting points of two adjacent rows consistent, resulting in displacement of the two adjacent rows; (3) the angular velocity of the sinusoidal characteristic during scanning is slow at the edges and fast in the middle, resulting in overall distortion with stretching at both ends and compression in the middle.

[0078] Phase inversion, shift and distortion cause the obtained image to be inconsistent with the actual shape of the object. If such an image is used in clinical practice, it will provide users with incorrect information, which will lead to incorrect diagnosis results, which is unacceptable. Therefore, the confocal endoscope must align the shift and correct the distortion to eliminate the problems caused by the scanning characteristics of the resonant mirror, provide users with correct images that are the same as the actual shape, and thus provide accurate diagnostic information for clinical practice. In order to solve at least some of the above problems, this application proposes a confocal endoscope image optimization method for performing image correction operations.

[0079] Please refer to FIG1 , which is a flow chart of a confocal endoscope image optimization method provided in an embodiment of the present application, which may specifically include:

[0080] S110, obtaining a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed;

[0081] Exemplarily, the data set to be calculated is obtained by performing a denoising operation on the data set to be processed, and the data set to be processed is obtained by performing a flipping operation on the original image data obtained by the confocal endoscope in the preprocessing mode, so that all images in the data set to be processed have the same acquisition direction.

[0082] S120, obtaining alignment parameters based on the data set to be calculated;

[0083] Illustratively, alignment parameters are extracted from the denoised dataset, and the alignment parameters are used to adjust the data points so that they are aligned in a specific arrangement.

[0084] S130, obtaining a correction target value based on the data set to be calculated;

[0085] Exemplarily, after the data are aligned, a correction target value is further calculated, which indicates the ideal state that each data point should achieve in the final optimized data set.

[0086] S140, performing an alignment operation on the sinusoidal sampling data point set based on the alignment parameters to obtain an aligned sinusoidal sampling data point set;

[0087] Exemplarily, the acquired alignment parameters are used to operate on the sinusoidal sampling data point set so as to align it with a predetermined standard or model, thereby obtaining an aligned sinusoidal sampling data point set.

[0088] S150 , performing a correction operation on the aligned sinusoidal sampling data point set based on the correction target value to obtain an optimized data set.

[0089] Exemplarily, finally, the aligned data point set is corrected according to the calculated correction target value to obtain an optimized data set.

[0090] It should be noted that the confocal endoscope can be set to a pre-processing mode and a normal operating mode. In the pre-processing mode, the data set to be calculated is acquired, and the alignment parameters and correction target values ​​are calculated based on the data set to be calculated. After the alignment parameters and correction target values ​​are calculated, they are stored in the corresponding storage unit. In the normal operating mode, the alignment and correction operations are performed on the collected images in real time by calling the alignment parameters and correction target values.

[0091] In summary, the present application provides a method for optimizing confocal endoscope images, which calculates alignment parameters and correction target values ​​in a preprocessing mode, performs alignment operations on image data based on the alignment parameters, and performs correction operations based on the correction target values. By switching between preprocessing and normal operating modes, the endoscope can adapt to different usage scenarios, ensuring that high-quality image data can be obtained under any conditions. This method can be used to align and correct the raw data collected by the confocal endoscope to eliminate the problems of data misalignment and distortion caused by the resonant mirror scanning process.

[0092] In some examples, the method further includes:

[0093] Selecting a first data set and two second data sets from the calculation data set, wherein each data in the second data set is adjacent to a data in the first data set in the calculation data set, and second data in different second data sets and in the same order as the first data set are located on both sides of the first data in the first data set in the calculation data set;

[0094] Performing an offset operation on the second data set based on the first data set, and calculating an alignment cost;

[0095] The target alignment parameters are determined based on the alignment cost values.

[0096] Exemplarily, a first data set and two second data sets are selected from the data sets to be calculated. Each data point in the second data set is spatially adjacent to the data point in the first data set, and the data points of the same order in each second data set are located on both sides of a specific data point in the first data set. Taking the first data set as a reference, the second data set is offset to achieve optimal alignment. The similarity or difference between the data sets can be evaluated by calculating the alignment cost value, which can be achieved through methods such as cross-correlation and Euclidean distance. This alignment cost value reflects the accuracy and effect of the alignment. Based on the calculated alignment cost value, the optimal alignment parameters are determined, such as offset, rotation angle or scaling factor. Find the alignment parameters that can minimize the difference between the first data set and the second data set to ensure the best consistency and correspondence between the data sets.

[0097] This embodiment provides a specific calculation method for alignment parameters. By selecting a suitable data set and performing spatial adjacency analysis, the refinement and efficiency of data processing are ensured. By calculating the alignment cost values ​​and adjusting the alignment parameters based on these values, the data sets can be accurately aligned, thereby ensuring the consistency and comparability of the data. The data sets that have been accurately aligned and optimized provide a solid foundation for subsequent image analysis and interpretation, thereby enhancing the reliability and validity of the analysis results. It can at least partially eliminate the problems caused by the scanning characteristics of the resonant mirror, provide users with correct images that are the same as the actual shape, and thus provide accurate diagnostic information for clinicians.

[0098] In some examples, selecting a first data set and two second data sets from the calculation data set includes:

[0099] Determine the starting row number and the ending row number of the target area in the image in the above calculation data set;

[0100] Perform segmentation based on the starting row number, the ending row number, and a preset number of segments to obtain the nearest odd or even row at the segment to form the first data set;

[0101] Second data adjacent to the first data in the first data set is acquired based on the first data set to form two second data sets.

[0102] For example, the method of selecting a first data set and two second data sets in the calculation data set is described in detail, taking the case where the first data set is an odd-numbered data set and the second data set is an even-numbered data set. The specific method of selecting P odd-numbered row numbers is to divide the image between the starting row (sl) and the ending row (el) into P+1 segments in the vertical direction, with a total of P segment intervals, and select the odd-numbered row number closest to each interval. The schematic diagram when P=3 is shown in Figure 5, which divides the scanning range into 4 equal segments. The 3 selected odd-numbered row numbers are the intervals between segments 1 and 2, segment 2 and segment 3, and segment 3 and segment 4, respectively.

[0103] Correspondingly, select two even-numbered row number sets SEL 1 and SEL 2 , each set contains P elements. SEL 1 The P even-numbered row numbers in the set are the corresponding P odd-numbered row numbers in SOL minus 1, SEL 2 The P even-numbered row numbers in the set are the corresponding P odd-numbered row numbers in SOL plus 1. p=0,1,…,P-1

[0104] Note that there are P odd rows in the SOL set. The data in OD is OD={od p},p=0,1,…,P-1

[0105] Remember SEL 1 There are P even rows in the set The data in

[0106] Remember SEL 2 There are P even rows in the set The data in

[0107] In summary, dividing the scan range into equal parts and selecting the nearest odd-numbered rows in each interval ensures spatial uniformity of data sampling, helps obtain representative data, and avoids sampling bias. By selecting adjacent odd and even rows, data comparison can be conveniently performed. This method simplifies the data processing process by systematically selecting data rows. The selection rules are clear and easy to implement, which is particularly important when processing large amounts of data. Because adjacent odd and even rows are selected, they are closely correlated in space. This close spatial correlation facilitates subsequent data alignment, especially in applications that require precise alignment of image data.

[0108] In some examples, determining the starting row number and the ending row number of the target region in the image in the calculation dataset includes:

[0109] Obtaining a pixel value histogram of the image data in the above-mentioned calculation data set;

[0110] Based on the above pixel value histogram, the OTSU algorithm is used to obtain the threshold information;

[0111] The starting row number and the ending row number are determined based on the threshold information.

[0112] For example, the image data in the computational data set is analyzed to obtain the frequency distribution of each pixel value, i.e., the pixel value histogram. The OTSU algorithm is an automatic threshold determination method that selects the optimal threshold by maximizing the inter-class variance. This threshold can divide the image into two parts: a valid area and an invalid area. Using the threshold obtained by the OTSU algorithm, it is possible to determine which row numbers of image data are important. Determining the starting row number and the ending row number is very important for focusing on a specific part of the image. In medical imaging, you may only be interested in a specific area of ​​the image, and this area can be determined by analyzing the pixel value histogram and applying the OTSU algorithm.

[0113] Specifically, the method for determining the starting line sl and the ending line el is: The pixel value histogram is statistically analyzed, and the result is shown in FIG6 . The OTSU algorithm is used on the above histogram to obtain the threshold T. The example histogram in FIG6 uses the OTSU algorithm to obtain the threshold T=88.

[0114] Use the following pseudo code to find sl:

[0115] The above pseudocode starts from the first row of the image and traverses to the last row. H is the total number of rows in the image. In each row, traverse from the second pixel to the second to last pixel. N is the total number of columns in the image. For each pixel in the currently traversed row, check whether the value of it and its left and right adjacent pixels are greater than a preset threshold T. This check is to identify continuous brightness changes in the horizontal direction, such changes may represent edges or feature lines. If a pixel sequence that meets the conditions is found in a certain row, this row is recorded as the starting row S. After recording the starting row, the entire traversal process ends and the starting row number is returned.

[0116] Use the following pseudo code to find el:

[0117] Start at the last row (H-1) of the image and traverse upward to the first row. Within each row, start at the second column and traverse to the second-to-last column. For each pixel D(i, j), check whether its value and the values ​​of the pixels on its left and right, D(i, j-1) and D(i, j+1), are all greater than a threshold T. If the above condition is met, record the row number i. Once a row number that meets the conditions is found, return that row number and terminate the program.

[0118] In summary, the methods proposed in the embodiments of this application use automated methods to determine the target region in an image, reducing the subjectivity and time cost of manual selection. The OTSU algorithm automatically determines the threshold through calculation, which is more efficient than manually adjusting the threshold. Based on statistical principles, the OTSU algorithm provides an objective method to determine the optimal threshold, thereby maintaining consistency across different image sets.

[0119] In some examples, the method further includes:

[0120] The alignment cost is calculated using the following formula:

[0121] Among them, P represents the number of lines used in the alignment operation, eid and sid represent the end line number and the start line number respectively, od p (id) represents the idth data point in the even row p under offset D, ed 1 p (id) and ed 2 p(id) represents the corresponding point of the idth data point in the odd row p in two different even row data sets, is a double summation symbol, indicating that all selected odd rows and each data point in these rows are to be iterated. and is the absolute value difference, representing the difference between the odd and even rows of a given data point under D offset, is the normalization factor.

[0122] For example, the offset is an integer, represented by a. Cost represents the alignment cost when the offset is a. When a takes integer values ​​in the range of [1-N1, N1-1] in sequence, the cost is calculated. Taking the odd-numbered row data as the benchmark, the even-numbered row data is offset according to a. The schematic diagram of different offset values ​​is shown in Figure 7, which are respectively the cases where the offset values ​​are -4, 0, and 3. od represents odd-numbered row data, and ed represents even-numbered row data. The numbers in the grid represent the subscripts of the data elements in each row of data.

[0123] Find the overlap between the odd and even rows after the offset. The overlap is represented by the odd row subscript. The starting and ending subscripts of the overlap are sid and eid respectively: sid = max(a,0) eid = min(N1-1,N1-1+a)

[0124] The alignment cost formula provided by the above formula provides a quantitative method to evaluate the effect of data alignment. By minimizing this cost function, the optimal alignment parameters can be found, thereby improving the accuracy and reliability of image processing tasks.

[0125] In some examples, the method further includes:

[0126] Obtain a uniform scanning sampling point data set and a sinusoidal sampling data point set;

[0127] Calculate the center difference information of the uniformly scanned sampling point data set and the sinusoidal sampling data point set;

[0128] Calculate the reference variable based on the above center value difference information, stretch coefficient, galvanometer scanning frequency and sine sampling frequency;

[0129] Determining a correction reference value based on the reference variable and the reference variable threshold information;

[0130] An approximate rounding operation is performed based on the above correction reference value and the sum of the central column information of the image in the sinusoidal sampling data point set to obtain a correction target value.

[0131] Exemplarily, two types of sampling data are collected: a uniform scanning sampling point dataset and a sinusoidal sampling data point dataset. The uniform sampling is derived from an ideal endoscopic image without scanning distortion, while the sinusoidal sampling dataset is a sampling data point dataset affected by the distortion caused by the movement of the galvanometer during the scanning process due to the reasons mentioned above. The uniform scanning dataset and the sinusoidal sampling dataset are compared, particularly their center positions, and the difference between the center points, i.e., the center difference information, is calculated. Such a difference can reflect any systematic deviations that may have occurred during the scanning process. The center difference information is used in conjunction with the stretch coefficient, the galvanometer scanning frequency, and the sinusoidal sampling frequency to calculate one or more reference variables. The reference variables are used to describe the scanning distortion pattern in the dataset and provide the necessary information for subsequent correction operations. The sum of the calculated correction reference value and the image center column information is used. This sum is approximately rounded to obtain a correction target value. The correction target value is used to adjust the image data in the sinusoidal sampling dataset. The correction operation may include translation, rotation, or other transformation of the data points in the image to compensate for the distortion or stretching caused by scanning.

[0132] The purpose of the embodiments of this application is to correct image distortion so that the image more accurately reflects the actual visual information. Through this method, each sampling point in the image is adjusted according to the calculated target value, thereby improving the overall image quality and reducing the impact of distortion. This correction process is usually automated, which can improve processing efficiency and ensure consistent and repeatable results.

[0133] In some examples, the reference variable threshold information includes first threshold information and second threshold information, the second threshold information is greater than the first threshold information, and the correction reference value includes a first correction reference value, a second correction reference value, and a third correction reference value.

[0134] The above-mentioned determination of the correction reference value based on the above-mentioned reference variable and the above-mentioned reference variable threshold information includes:

[0135] When the reference variable is less than the first threshold value information, the first correction reference value is determined as the correction reference value; or, when the reference variable is greater than the second threshold value information, the second correction reference value is determined as the correction reference value:

[0136] The first correction reference value dn is determined based on the following formula 11 :

[0137] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency;

[0138] The second correction value dn is determined based on the following formula 12 :

[0139] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency;

[0140] The third correction value dn is determined based on the following formula: 13 :

[0141] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sine sampling frequency, α is the stretching coefficient, dn2 is the center difference information, and arcsin() is the inverse sine function.

[0142] Exemplarily, the first correction reference value is calculated by the galvanometer scanning frequency and the sinusoidal sampling frequency, and they are reciprocals of each other. The third correction reference value is determined by the arcsine value related to the galvanometer scanning frequency, the sinusoidal sampling frequency and the product of the stretch coefficient, the center difference information, the galvanometer scanning frequency and the sinusoidal sampling frequency.

[0143] In some examples, the method further includes:

[0144] Performing a binarization operation on the image data to be calculated in the above-mentioned data set to be calculated to obtain a binary image set;

[0145] performing a morphological closing operation on the binary image data in the binary image set to obtain a closed image set;

[0146] Acquire height information, width information, and edge column information of the elliptical region in the closed image in the closed image set;

[0147] Correction parameters are calculated based on the height information, width information, and outermost column information, wherein the correction parameters include stretch coefficient information and center column information.

[0148] As an example, FIG8 shows a schematic diagram of an image to be calculated according to an embodiment of the present application. The image dataset to be calculated is an image dataset to be corrected. The pixel values ​​in the image are simplified to two possible values, typically 0 and 1, to separate foreground objects from background, i.e., valid image areas from invalid image areas, thereby forming a binary image as shown in FIG9 . By selecting an appropriate threshold, pixel values ​​above the threshold are set to 1, typically representing the foreground, and pixels below the threshold are set to 0, representing the background. This improves the continuity of objects in the binary image, filling holes and breaks within objects for clearer boundaries. First, a dilation operation is performed on the image to enlarge the foreground objects; then an erosion operation is performed to restore the objects to their original size while preserving the filling effect. This results in a closed image set, as shown in FIG10 , a closed image provided in this embodiment, in which the foreground object boundaries are closed, without holes or breaks. As shown in FIG10 , objects resembling elliptical shapes are extracted from the closed image, and their size information is obtained. Objects in the closed image are analyzed, their heights and widths are measured, and their positions within the image, i.e., their edge column information, are determined. The collected height, width, and outermost column information will be used to calculate correction parameters in subsequent steps. The extracted size information is used to generate image correction parameters, including a stretch factor and information about the image's center column. Correction parameters, such as the stretch factor, are calculated using a mathematical model based on the physical dimensions of the elliptical region and the pixel size in the image to correct for scale distortion. The resulting stretch factor and center column information can then be used to adjust the entire image set to improve image quality.

[0149] In some examples, the method further includes:

[0150] Calculate the stretch coefficient information according to the height information and the width information;

[0151] The above center column information C is calculated according to the following formula:

[0152] Where C is the coordinate of the geometric center point of the image, c0 is the edge column information, H1 is the height information, W1 is the width information, and f sample is the sampling frequency, f scan is the scanning frequency, and arcsin() is the inverse sine function.

[0153] For example, the stretch coefficient information α can be calculated from the height information H1 and the width information W1, and can be specifically calculated according to the following formula:

[0154] For example, the correction parameter C can be used to correct distortions that may be introduced during the image scanning process, such as curved scan lines or deformation caused by nonlinear scanning speeds. This correction yields more accurate results for subsequent processing such as feature extraction and edge detection. This correction method is not limited to specific image types and is applicable to a variety of image data acquired by scanning devices.

[0155] In some examples, a row of input data is denoted as vals1[N1], and the corrected data for that row is vals2[N2]. As shown in Figure 11, the algorithm associates the uniform scan sampling point subscript n2 with a sinusoidal sampling point subscript n1, and the brightness value of the uniform scan sampling point subscript n2 is equal to the brightness value of the sinusoidal sampling point subscript n1. Figure 11 is a schematic diagram of the principle of a correction algorithm provided in an embodiment of the present application, and Figure 12 is a schematic diagram of an uncorrected image provided in an embodiment of the present application. Figure 13 is a schematic diagram of an overcorrected image provided in an embodiment of the present application.

[0156] This can be achieved by the following correction algorithm 1 and correction algorithm 2:

[0157] The pseudo code of correction algorithm 1 is as follows:

[0158] enter:

[0159] vals1[N1]——sine sweep signal data

[0160] f scan ——Galvanometer scanning frequency, unit Hz;

[0161] f sample ——Sine sampling frequency, unit Hz;

[0162] C – center column;

[0163] α——stretching coefficient;

[0164] Output:

[0165] vals2[N2]——uniform scanning signal data

[0166] step:

[0167] It should be noted that the round function in line 12 is an approximate rounding function.

[0168] The main steps of the correction algorithm 1 include the following:

[0169] 1. Initialize variables:

[0170] vals1[N]: Array storing the initial data set.

[0171] f scan : Scanning frequency, in Hz.

[0172] f sample : Sampling frequency, in Hz.

[0173] C: Center column.

[0174] α: stretching coefficient, used to adjust the correction amplitude.

[0175] 2. Calculate intermediate variables:

[0176] vals2[N]: Array to store corrected data.

[0177] C2: Calculated as half the length of the vals1 array, used to determine the center position of the data set.

[0178] 3. Loop through each data point:

[0179] For n from 0 to N-1 (the length of the dataset), calculate the distortion d for each point n2 .

[0180] d n2 Calculated as n minus C2, d n2 To characterize the offset relative to the center of the dataset.

[0181] 4. According to d n2 Choose different correction methods for the value of:

[0182] The first threshold information can be -1, and the second threshold information can be 1. When the reference variable is less than -1, the reference variable is: When the above-mentioned reference variable is less than the above-mentioned first threshold information, the above-mentioned first correction reference value is determined as the correction reference value; when the above-mentioned reference variable is greater than the above-mentioned second threshold information, the above-mentioned second correction reference value is determined as the correction reference value; when the above-mentioned reference variable is greater than or equal to the above-mentioned first threshold information and less than or equal to the above-mentioned second threshold information, the above-mentioned third correction reference value is determined as the correction reference value.

[0183] The first correction reference value dn is determined based on the following formula 11 :

[0184] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency.

[0185] The second correction value dn is determined based on the following formula 12 :

[0186] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency.

[0187] The third correction value dn is determined based on the following formula: 13 :

[0188] Among them, f scan is the scanning frequency of the galvanometer, f sample is the sine sampling frequency, α is the stretching coefficient, dn2 is the center difference information, and arcsin() is the inverse sine function.

[0189] 5. Calculate and correct the index:

[0190] Use the round function to round the result of C+d1 to get the corrected index n1;

[0191] If n1 is less than 0 or greater than N-1, it is set to the value of the first or last element of the array vals1, respectively, to prevent the index from going out of bounds.

[0192] If d n1 The corresponding values ​​are directly obtained from vals1 within the valid range. The algorithm effectively assigns a corrected value to each data point, thereby generating a new dataset vals2 that reflects the corrected sampled data.

[0193] The pseudo code of correction algorithm 2 is as follows:

[0194] enter:

[0195] vals1[N1]——sine sweep signal data

[0196] f scan ——Galvanometer scanning frequency, unit Hz;

[0197] f sample ——Sine sampling frequency, unit Hz;

[0198] C – center column;

[0199] α——stretching coefficient;

[0200] Output:

[0201] vals2[N2]——uniform scanning signal data

[0202] step:

[0203] The main steps of the correction algorithm 2 include the following:

[0204] 1. Initialize variables:

[0205] vals1[N]: Array storing the initial data set.

[0206] f scan : Scanning frequency, in Hz.

[0207] f sample : Sampling frequency, in Hz.

[0208] C: Center column.

[0209] α: stretching coefficient, used to adjust the correction amplitude.

[0210] 2. Calculate intermediate variables:

[0211] vals2[N]: Array to store corrected data.

[0212] C2: Calculated as half the length of the vals1 array, used to determine the center position of the data set.

[0213] 3. Loop through each data point:

[0214] For n from 0 to N-1 (the length of the dataset), calculate the distortion d for each point n2 .

[0215] d n2 Calculated as n minus C2, d n2 To characterize the offset relative to the center of the dataset.

[0216] 4. According to d n2 Choose different correction methods for the value of :

[0217] The first threshold information can be -1, and the second threshold information can be 1. When the reference variable is less than -1, the reference variable is: When the above-mentioned reference variable is less than the above-mentioned first threshold information, the above-mentioned first correction reference value is determined as the correction reference value; when the above-mentioned reference variable is greater than the above-mentioned second threshold information, the above-mentioned second correction reference value is determined as the correction reference value; when the above-mentioned reference variable is greater than or equal to the above-mentioned first threshold information and less than or equal to the above-mentioned second threshold information, the above-mentioned third correction reference value is determined as the correction reference value.

[0218] Please refer to FIG14 , an embodiment of the confocal endoscope image optimization device in the embodiment of the present application may include:

[0219] A first acquisition unit 21 is configured to acquire a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed;

[0220] A second acquiring unit 22 is configured to acquire alignment parameters based on the data set to be calculated;

[0221] A third acquiring unit 23 is configured to acquire a correction target value based on the data set to be calculated;

[0222] a fourth acquiring unit 24, configured to perform an alignment operation on the sinusoidal sampled data point set based on the alignment parameter to acquire an aligned sinusoidal sampled data point set;

[0223] The fifth acquiring unit 25 is configured to perform a correction operation on the aligned sinusoidal sampling data point set based on the correction target value to acquire an optimized data set.

[0224] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0225] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0226] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flow chart and / or block diagram and the combination of the process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes and / or one box or multiple boxes of the flow chart.

[0227] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0229] The present application also provides a computer program product comprising computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the confocal endoscope image optimization process in the corresponding embodiment.

[0230] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0231] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0233] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0235] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0236] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application. Industrial Applicability

[0237] The confocal endoscope image optimization method and related equipment provided in each embodiment of the present application calculate the alignment parameters and correction target values ​​in the preprocessing mode, and in the normal working mode, perform alignment and correction operations on the original image data collected by the confocal endoscope in real time by calling the alignment parameters and correction target values ​​to eliminate the problems of data misalignment and distortion caused by the resonant mirror scanning process, and can ensure that the endoscope can obtain high-quality image data under any conditions. The confocal endoscope image optimization method and related equipment provided in each embodiment of the present application are easy to manufacture and use, and have industrial applicability.

Claims

1. A confocal endoscope image optimization method, It is characterized in that include: Acquire a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed; Acquire alignment parameters based on the data set to be calculated; Acquire a correction target value based on the data set to be calculated; Performing an alignment operation on the sinusoidal sampling data point set based on the alignment parameters to obtain an aligned sinusoidal sampling data point set; A correction operation is performed on the aligned sinusoidal sampling data point set based on the correction target value to obtain an optimized data set.

2. The confocal endoscope image optimization method according to claim 1, It is characterized in that Also includes: Selecting a first data set and two second data sets from the calculation data set, wherein each data in the second data set is adjacent to data in the first data set in the calculation data set, and second data in the same order as the first data set in different second data sets are located on both sides of the first data in the first data set in the calculation data set; An offset operation is performed on the second data set based on the first data set, and an alignment cost value is calculated; and a target alignment parameter is determined based on the alignment cost value.

3. The confocal endoscope image optimization method according to claim 2, It is characterized in that The selecting a first data set and two second data sets from the calculation data set comprises: Determine the starting row number and the ending row number of the target area in the image in the calculation data set; Perform segmentation processing based on the starting row number, the ending row number and the preset number of segments to obtain the nearest odd-numbered row or even-numbered row at the segment to form the first data set; Second data adjacent to the first data in the first data set is acquired based on the first data set to form two second data sets.

4. The confocal endoscope image optimization method according to claim 3, It is characterized in that The determining the starting row number and the ending row number of the target area in the image in the calculation data set includes: Obtaining a pixel value histogram of the image data in the calculation data set; Obtaining threshold information using the OTSU algorithm based on the pixel value histogram; The start row number and the end row number are determined based on the threshold information.

5. The confocal endoscope image optimization method according to claim 2, It is characterized in that Also includes: The alignment cost is calculated by the following formula: Among them, P represents the number of lines used in the alignment operation, eid and sid represent the end line number and the start line number respectively, and od p (id) represents the idth data point in the even row p under offset D, and They represent the corresponding points of the idth data point in the odd row p in two different even row data sets, is a double summation symbol, indicating that all selected odd rows and each data point in these rows are to be iterated. and is the absolute value difference, representing the difference between the odd and even rows of a given data point under the D offset, is the normalization factor.

6. The confocal endoscope image optimization method according to claim 1, It is characterized in that Also includes: Obtain a uniform scanning sampling point data set and a sinusoidal sampling data point set; Calculate the central difference information of the uniform scanning sampling point data set and the sinusoidal sampling data point set; Calculate reference variables based on the center value difference information, stretch coefficient, galvanometer scanning frequency and sine sampling frequency; determining a correction reference value based on the reference variable and the reference variable threshold information; An approximate rounding operation is performed based on the correction reference value and the sum of the central column information of the image in the sinusoidal sampling data point set to obtain a correction target value.

7. The confocal endoscope image optimization method according to claim 6, It is characterized in that The reference variable threshold information includes first threshold information and second threshold information, the second threshold information is greater than the first threshold information, and the correction reference value includes a first correction reference value, a second correction reference value and a third correction reference value, The determining of the correction reference value based on the reference variable and the reference variable threshold information comprises: In the case where the reference variable is less than the first threshold information, the first correction reference value is determined as the correction reference value; or, in the case where the reference variable is greater than the second threshold information, the second correction reference value is determined as the correction reference value: The first correction reference value dn is determined based on the following formula 11 : Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency; The second correction value dn is determined based on the following formula 12 : Among them, f scan is the scanning frequency of the galvanometer, f sample is the sinusoidal sampling frequency; The third correction value dn is determined based on the following formula 13 : Among them, f scan is the scanning frequency of the galvanometer, f sample is the sine sampling frequency, α is the stretching coefficient, dn 2 is the center difference information, and arcsin() is the inverse sine function.

8. The confocal endoscope image optimization method according to claim 5, It is characterized in that Also includes: Performing a binarization operation on the image data to be calculated in the data set to be calculated to obtain a binary image set; Performing a morphological closing operation on the binary image data in the binary image set to obtain a closed image set; Acquire height information, width information and edge column information of the elliptical region in the closed image in the closed image set; Correction parameters are calculated based on the height information, the width information, and the outermost column information, wherein the correction parameters include stretch factor information and center column information.

9. The confocal endoscope image optimization method according to claim 7, It is characterized in that Also includes: Calculate the stretch factor information according to the height information and the width information; The center column information C is calculated according to the following formula: Where C is the coordinate of the geometric center point of the image, c 0 is the most marginal information, H 1 is the height information, W 1 is the width information, f sample is the sampling frequency, f scan is the scanning frequency, arcsin() is the inverse sine function.

10. A confocal endoscope image optimization device, It is characterized in that include: A first acquisition unit is used to acquire a data set to be calculated, wherein the data set to be calculated is obtained by performing a data denoising operation on the data to be processed; A second acquisition unit, configured to acquire an alignment parameter based on the data set to be calculated; A third acquisition unit, configured to acquire a correction target value based on the data set to be calculated; a fourth acquisition unit, configured to perform an alignment operation on the sinusoidal sampling data point set based on the alignment parameter to acquire an aligned sinusoidal sampling data point set; A fifth acquisition unit is used to perform a correction operation on the aligned sinusoidal sampling data point set based on the correction target value to obtain an optimized data set.

Images