Endoscope image display method, apparatus, storage medium, and device

Abstract

The present disclosure relates to an endoscope image display method, device, storage medium and equipment, comprising: in response to the received specified image coordinates of the specified endoscope image of a specific position, determining the target image coordinates of the specified image coordinates from the positioning coordinates corresponding to the endoscope image; according to the index relationship between the advancing coordinates of the endoscope and the valuable endoscope image, querying the multiple target endoscope images corresponding to the target image coordinates from the image database, and the image database saves multiple valuable endoscope images corresponding to each advancing coordinate; image equalization processing and repair processing are performed on each target endoscope image, and an endoscope display image is generated by fusion; the endoscope display image is displayed on the viewport of the human-computer interaction interface. Image equalization processing and repair processing are performed on multiple valuable endoscope images, and an endoscope display image is generated by fusion, and then displayed on the viewport of the human-computer interaction interface, which can improve the accuracy of image display.

Description

Technical Field

[0001] This disclosure relates to the field of image storage and display technology, and more specifically, to an endoscopic image display method, apparatus, storage medium, and device. Background Technology

[0002] Endoscopes can capture images of the human body, helping doctors determine the location and extent of lesions in tissues and organs. During surgery, they also aid in positioning surgical instruments precisely within the body. However, all endoscopic images are saved, resulting in a large data volume, and many images cannot be highlighted during presentation. Furthermore, uneven exposure during endoscopy can lead to variations in brightness and color, causing inaccuracies in the displayed images. Summary of the Invention

[0003] To address the aforementioned technical problems, the purpose of this disclosure is to provide an endoscopic image display method, apparatus, storage medium, and device.

[0004] To achieve the above objectives, a first aspect of this disclosure provides an endoscopic image display method, comprising:

[0005] In response to receiving specified image coordinates of an endoscope image to be displayed at a specific location, the target image coordinates of the specified image coordinates are determined from the set of positioning coordinates corresponding to the endoscope image;

[0006] Based on the index relationship between the endoscope's advancement coordinates and the valuable endoscope images, multiple target endoscope images corresponding to the target image coordinates are queried from the image database. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate.

[0007] The target endoscopic images are subjected to image equalization and repair processing, and then fused to generate an endoscopic display image;

[0008] The endoscopic image is displayed in the viewport of the human-computer interaction interface.

[0009] Optionally, the step of performing image equalization and repair processing on each of the target endoscopic images, and fusing them to generate an endoscopic display image, includes:

[0010] The target endoscope image is subjected to brightness and color equalization processing to obtain a first backup endoscope image;

[0011] Glare repair is performed on the first backup endoscope image to obtain the second backup endoscope image;

[0012] The second backup endoscope image is transformed from the Cartesian coordinate system to the polar coordinate system to obtain a polar coordinate endoscope image;

[0013] Image feature point detection is performed on the polar coordinate endoscope image to obtain endoscope image feature points;

[0014] Based on the RANSAC algorithm, normal and abnormal feature points in the endoscopic image feature points are determined;

[0015] The abnormal feature points are removed from the feature points of the endoscopic image to obtain an endoscopic display image that retains only the normal feature points.

[0016] Optionally, the step of removing the abnormal feature points from the endoscopic image feature points to obtain an endoscopic display image that retains only the normal feature points includes:

[0017] The region where the normal feature points are located in each of the endoscopic display images is taken as the endoscopic image region to be fused for the corresponding endoscopic display images;

[0018] Based on the Gaussian pyramid algorithm, convolution and downsampling are performed on each of the endoscopic image regions to be fused to obtain Gaussian pyramid endoscopic images with multiple different resolution scales corresponding to each of the endoscopic image regions to be fused.

[0019] Based on the Laplacian pyramid algorithm, the corresponding Laplacian endoscopic image is generated by upsampling from the bottom layer image of each Gaussian pyramid endoscopic image.

[0020] Calculate the average value of the pixels corresponding to each layer in each of the Laplacian endoscope images;

[0021] Image layers with the same average pixel value are used as a common endoscope image layer. The Laplacian endoscope images are then fused to obtain the endoscope display image.

[0022] Optionally, the multiple valuable endoscopic images corresponding to each of the aforementioned advancement coordinates are determined in the following manner:

[0023] Each endoscope image corresponding to the advancement coordinate is divided into a grid to obtain the endoscope image corresponding to each grid.

[0024] The contrast index of each grid is determined based on the edge detection operator of the endoscopic image of each grid.

[0025] The quality index corresponding to each grid is determined based on the brightness of the endoscopic image of each grid.

[0026] The overall quality of each endoscope image is determined based on the contrast index and the quality index corresponding to each grid of each endoscope image;

[0027] Based on the overall quality, valuable endoscopic images are determined from the plurality of endoscopic images.

[0028] Optionally, the step of determining the quality index corresponding to each grid based on the brightness of the endoscopic image of each grid includes:

[0029] Construct a brightness histogram for each grid based on the brightness of the endoscopic image corresponding to each grid.

[0030] Based on preset selection parameters, the target bin is determined from the bins of each brightness histogram;

[0031] Determine the number of pixels contained in each bin of the target bin, and the number of pixels contained in each bin of the brightness histogram;

[0032] The quality index corresponding to each grid is determined based on the weight value of each bin in the brightness histogram, the number of pixels contained in each bin in the brightness histogram, and the number of pixels contained in each bin corresponding to each target bin.

[0033] Optionally, the step of determining the contrast index of each grid based on the edge detection operator of the endoscopic image of each grid includes:

[0034] Edge detection is performed on the endoscope image corresponding to each grid, and the absolute value of the edge detection operator for each pixel coordinate in each endoscope image is calculated;

[0035] The absolute values ​​of the edge detection operators corresponding to the pixel coordinates of each grid are summed to obtain the edge detection sum value of each grid.

[0036] The contrast index corresponding to each grid is determined based on the grid weight corresponding to each grid and the edge detection value.

[0037] Optionally, the step of determining the overall quality of each endoscope image based on the contrast index and the quality index corresponding to each grid of each endoscope image includes:

[0038] The maximum contrast index corresponding to each of the endoscopic images is taken as the target contrast index, and the maximum quality index corresponding to each of the endoscopic images is taken as the target quality index.

[0039] The sum of the contrast index values ​​is obtained by multiplying the exponential weight of the contrast index by the contrast index of each grid of the endoscopic image, dividing by the quotient of the target contrast index, and then summing the results.

[0040] The sum of the product of the index weight of the quality index and the quality index of each grid of the endoscopic image, divided by the quotient of the target quality index, is used to obtain the quality index sum value.

[0041] The sum of the contrast index and the quality index is used to determine the overall quality of the endoscopic image.

[0042] A second aspect of this disclosure provides an endoscopic image display device, comprising:

[0043] The determination module is configured to, in response to receiving specified image coordinates of an endoscope image at a specified location, determine the target image coordinates of the specified image coordinates from a set of positioning coordinates corresponding to the endoscope image;

[0044] The query module is configured to query multiple target endoscope images corresponding to the target image coordinates from the image database based on the index relationship between the endoscope's advancement coordinates and the valuable endoscope images. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate.

[0045] The fusion generation module is configured to perform image equalization and repair processing on each of the target endoscope images, and fuse them to generate an endoscope display image;

[0046] The display module is configured to display the endoscope image in the viewport of the human-computer interaction interface.

[0047] Optionally, the fusion generation module is configured as follows:

[0048] The target endoscope image is subjected to brightness and color equalization processing to obtain a first backup endoscope image;

[0049] Glare repair is performed on the first backup endoscope image to obtain the second backup endoscope image;

[0050] The second backup endoscope image is transformed from the Cartesian coordinate system to the polar coordinate system to obtain a polar coordinate endoscope image;

[0051] Image feature point detection is performed on the polar coordinate endoscope image to obtain endoscope image feature points;

[0052] Based on the RANSAC algorithm, normal and abnormal feature points in the endoscopic image feature points are determined;

[0053] The abnormal feature points are removed from the feature points of the endoscopic image to obtain an endoscopic display image that retains only the normal feature points.

[0054] Optionally, the fusion generation module is configured as follows:

[0055] The region where the normal feature points are located in each of the endoscopic display images is taken as the endoscopic image region to be fused for the corresponding endoscopic display images;

[0056] Based on the Gaussian pyramid algorithm, convolution and downsampling are performed on each of the endoscopic image regions to be fused to obtain Gaussian pyramid endoscopic images with multiple different resolution scales corresponding to each of the endoscopic image regions to be fused.

[0057] Based on the Laplacian pyramid algorithm, the corresponding Laplacian endoscopic image is generated by upsampling from the bottom layer image of each Gaussian pyramid endoscopic image.

[0058] Calculate the average value of the pixels corresponding to each layer in each of the Laplacian endoscope images;

[0059] Image layers with the same average pixel value are used as a common endoscope image layer. The Laplacian endoscope images are then fused to obtain the endoscope display image.

[0060] Optionally, the endoscopic image display device includes: a storage module configured to:

[0061] Each endoscope image corresponding to the advancement coordinate is divided into a grid to obtain the endoscope image corresponding to each grid.

[0062] The contrast index of each grid is determined based on the edge detection operator of the endoscopic image of each grid.

[0063] The quality index corresponding to each grid is determined based on the brightness of the endoscopic image of each grid.

[0064] The overall quality of each endoscope image is determined based on the contrast index and the quality index corresponding to each grid of each endoscope image;

[0065] Based on the overall quality, valuable endoscopic images are determined from the plurality of endoscopic images.

[0066] Optionally, the storage module is configured as follows:

[0067] Construct a brightness histogram for each grid based on the brightness of the endoscopic image corresponding to each grid.

[0068] Based on preset selection parameters, the target bin is determined from the bins of each brightness histogram;

[0069] Determine the number of pixels contained in each bin of the target bin, and the number of pixels contained in each bin of the brightness histogram;

[0070] The quality index corresponding to each grid is determined based on the weight value of each bin in the brightness histogram, the number of pixels contained in each bin in the brightness histogram, and the number of pixels contained in each bin corresponding to each target bin.

[0071] Optionally, the storage module is configured as follows:

[0072] Edge detection is performed on the endoscope image corresponding to each grid, and the absolute value of the edge detection operator for each pixel coordinate in each endoscope image is calculated;

[0073] The absolute values ​​of the edge detection operators corresponding to the pixel coordinates of each grid are summed to obtain the edge detection sum value of each grid.

[0074] The contrast index corresponding to each grid is determined based on the grid weight corresponding to each grid and the edge detection value.

[0075] Optionally, the storage module is configured as follows:

[0076] The maximum contrast index corresponding to each of the endoscopic images is taken as the target contrast index, and the maximum quality index corresponding to each of the endoscopic images is taken as the target quality index.

[0077] The sum of the contrast index values ​​is obtained by multiplying the exponential weight of the contrast index by the contrast index of each grid of the endoscopic image, dividing by the quotient of the target contrast index, and then summing the results.

[0078] The sum of the product of the index weight of the quality index and the quality index of each grid of the endoscopic image, divided by the quotient of the target quality index, is used to obtain the quality index sum value.

[0079] The sum of the contrast index and the quality index is used to determine the overall quality of the endoscopic image.

[0080] A third aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a first processor, implements the steps of the endoscopic image display method according to any one of the first aspects.

[0081] A fourth aspect of this disclosure provides an electronic device, comprising:

[0082] The first memory, on which the computer program is stored;

[0083] A second processor is configured to execute the computer program in the first memory to implement the steps of the endoscopic image display method according to any one of the first aspects.

[0084] The above technical solution can achieve at least the following beneficial effects:

[0085] In response to received specified image coordinates of an endoscopic image at a designated location, the system determines the target image coordinates from the set of positioning coordinates corresponding to the endoscopic images. Based on the index relationship between the endoscope's advancement coordinates and valuable endoscopic images, it queries an image database for multiple valuable endoscopic images corresponding to the target image coordinates. The image database stores multiple valuable endoscopic images corresponding to each advancement coordinate. Image equalization and repair processing are performed on each target endoscopic image, and the images are then fused to generate an endoscopic display image. This endoscopic display image is then displayed in the viewport of the human-computer interaction interface. Performing image equalization and repair processing on multiple valuable endoscopic images, fusing them to generate an endoscopic display image, and then displaying it in the viewport of the human-computer interaction interface improves the accuracy of image display.

[0086] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0087] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0088] Figure 1 This is a flowchart illustrating an endoscopic image display method according to an embodiment of the present disclosure.

[0089] Figure 2 This is a flowchart illustrating an endoscopic image storage method according to an embodiment of the present disclosure.

[0090] Figure 3 This is an implementation shown according to an embodiment of the present disclosure. Figure 2 The flowchart for step S16.

[0091] Figure 4 This is an implementation shown according to an embodiment of the present disclosure. Figure 3 The flowchart for step S162.

[0092] Figure 5 This is an implementation shown according to an embodiment of the present disclosure. Figure 2The flowchart for step S12.

[0093] Figure 6 This is a block diagram of an endoscopic image display device according to an embodiment of the present disclosure.

[0094] Figure 7 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0095] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0096] Before introducing the endoscopic image display method, apparatus, storage medium, and device provided in this disclosure, the application scenario of this disclosure will be described first. If all images acquired by the endoscope are saved, not only will the amount of data acquired during the procedure be large, but the amount of data of the saved endoscopic images will also be large. Furthermore, during the display of endoscopic images, the images captured due to uneven exposure time may have deviations in brightness and color, resulting in certain deviations in the image presentation.

[0097] Therefore, this disclosure provides an endoscopic image display method to eliminate deviations caused by uneven exposure time and reduce the amount of image acquisition and data storage required by the endoscope. See [link to relevant documentation]. Figure 1 As shown, the method includes the following steps.

[0098] In step S1, in response to the received specified image coordinates of the endoscope image to be displayed at a specific location, the target image coordinates of the specified image coordinates are determined from the set of positioning coordinates corresponding to the endoscope image.

[0099] In step S1, in response to the received specified image coordinates of the endoscope image to be displayed at a specific location, the advance distance between the specified image coordinates and each positioning coordinate in the positioning coordinate set corresponding to the endoscope image can be calculated based on the advance path of the endoscope when acquiring the endoscope image.

[0100] The advancing distance is the distance from the specified image coordinates to the positioning coordinates, or the advancing distance is the distance from the positioning coordinates to the specified image coordinates.

[0101] The positioning coordinate with the smallest advancing distance in the positioning coordinate set is determined as the target image coordinate of the specified image coordinate.

[0102] In step S2, based on the index relationship between the endoscope's advancement coordinates and the valuable endoscope images, multiple target endoscope images corresponding to the target image coordinates are queried from the image database. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate.

[0103] In this disclosure, when a doctor issues a control command through a human-computer interaction interface to request the replay of historical images of a specific location, the system finds the closest positioning coordinates from the set of positioning coordinates based on the specified image coordinates input by the doctor, and then finds multiple target endoscopic images that have been saved for the positioning coordinates from the image database.

[0104] In step S3, image equalization and repair processing are performed on each target endoscope image, and the images are then fused to generate an endoscope display image.

[0105] Optionally, the steps of performing image equalization and restoration processing on each target endoscopic image, and fusing them to generate an endoscopic display image, include:

[0106] The target endoscope image is processed for brightness and color equalization to obtain the first backup endoscope image.

[0107] In this embodiment of the disclosure, brightness and color equalization processing can reduce the interference to the image processing algorithm caused by the time non-uniformity of endoscope exposure and the time non-uniformity of endoscope illumination source.

[0108] Glare restoration was performed on the first backup endoscope image to obtain the second backup endoscope image.

[0109] In this embodiment of the disclosure, glare repair can reduce the interference of local highlights in the image caused by reflections from human tissue on the image processing algorithm.

[0110] The second backup endoscope image is transformed from the Cartesian coordinate system to the polar coordinate system to obtain the polar coordinate endoscope image.

[0111] Image feature points are detected by performing image feature point detection on polar coordinate endoscope images to obtain endoscope image feature points.

[0112] In this embodiment of the disclosure, feature points in an image can be detected using algorithms such as SIFT and SURF.

[0113] Based on the RANSAC algorithm, normal and abnormal feature points in endoscopic images are identified.

[0114] This allows for the filtering of feature points, retaining high-quality feature points and eliminating unreliable feature points.

[0115] Abnormal feature points are removed from the feature points of the endoscopic image to obtain an endoscopic display image that retains only normal feature points.

[0116] Optionally, the step of removing abnormal feature points from the endoscopic image feature points to obtain an endoscopic display image that retains only normal feature points includes:

[0117] The region where the normal feature points are located in each endoscopic image is taken as the endoscopic image region to be fused in the corresponding endoscopic image.

[0118] Based on the Gaussian pyramid algorithm, convolution and downsampling are performed on each endoscopic image region to be fused to obtain Gaussian pyramid endoscopic images with multiple different resolution scales corresponding to each endoscopic image region to be fused.

[0119] Based on the Laplacian pyramid algorithm, the corresponding Laplacian endoscopic image is generated by upsampling from the bottom layer image of each Gaussian pyramid endoscopic image.

[0120] Calculate the average value of the pixels corresponding to each layer in each Laplacian endoscope image.

[0121] Image layers with the same average pixel value are used as a common endoscopic image layer. Image fusion is performed on each Laplacian endoscopic image to obtain the endoscopic display image.

[0122] In step S4, the endoscopic image is displayed in the viewport of the human-computer interaction interface.

[0123] The above technical solution responds to the received specified image coordinates of an endoscope image intended to display a specific location. It determines the target image coordinates from the set of positioning coordinates corresponding to the endoscope image; based on the index relationship between the endoscope's advancement coordinates and valuable endoscope images, it queries an image database for multiple valuable endoscope images corresponding to the target image coordinates. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate; it performs image equalization and repair processing on each target endoscope image, and then fuses them to generate an endoscope display image; finally, it displays the endoscope display image in the viewport of the human-computer interaction interface. Performing image equalization and repair processing on multiple valuable endoscope images, fusing them to generate an endoscope display image, and then displaying it in the viewport of the human-computer interaction interface can improve the accuracy of image display.

[0124] This disclosure also provides a method for saving endoscopic images, see [link to relevant documentation]. Figure 2 The method and steps are shown.

[0125] In step S11, the planned advancement path of the endoscope is obtained.

[0126] In this embodiment of the disclosure, the planned advancement path can be a preset advancement path for the current image acquisition before the endoscope is advanced. For example, the approximate location of the lesion can be inferred from X-ray or B-ultrasound, and then the planned advancement path for advancing the endoscope can be obtained based on the starting point of the endoscope advancement and the approximate location of the lesion.

[0127] In step S12, the planned advancement path is divided into multiple planned advancement sub-paths, where each planned advancement sub-path is approximately a straight line path.

[0128] In one embodiment of this disclosure, a curve mathematical model can be performed on the planned advancement path. For example, a polynomial function can be established for approximation to obtain each planned advancement sub-path that is approximately a straight line. In specific implementation, the planned path is segmented using stored mathematical model parameters. The space required for storing parameters is small, and only fitting calculations are needed for all coordinates on the planned advancement path.

[0129] In another embodiment of this disclosure, the planned advancement path is discretized into a curve, and a set of control points is used to replace the planned advancement path. This allows the coordinate values ​​of the control points to be read directly to complete the segmentation without complex calculations.

[0130] In another embodiment of this disclosure, a sampling interval parameter can be selected from the control parameter set to divide the planned advancement path into straight segments of equal length. For example, after experimentation, the sampling interval parameter can be set to 10mm, that is, the planned advancement path can be divided into multiple planned advancement sub-paths of 10mm length. The sampling interval parameter of 10mm can well adapt to the natural curvature of various human tissues.

[0131] It should be noted that there is no strict limit to the length requirement of each planned advancement sub-path. The aforementioned 10mm is only a preferred method, and the division does not have to be of equal length. It is permissible for the actual length of some planned advancement sub-paths to be slightly longer or shorter than the average. If the system's default sampling interval parameter is not suitable for certain scenarios, users can be allowed to change it to other values.

[0132] In step S13, for each planned advancement sub-path, the positioning coordinates of the planned advancement sub-path are determined.

[0133] In this embodiment of the disclosure, the positioning coordinates of each planned advancement sub-path can be specified, for example, the midpoint of each planned advancement sub-path can be specified as the positioning coordinates. Of course, other coordinates with obvious characteristics on the planned advancement sub-path can also be used as positioning coordinates, with the same effect. For example, the location where the space narrows on the planned advancement sub-path.

[0134] In step S14, a set of positioning coordinates corresponding to the endoscopic images is constructed based on the positioning coordinates corresponding to each planned advancement sub-path.

[0135] Understandably, the set of positioning coordinates includes all the positioning coordinates of the planned advancement path for this endoscopic image acquisition.

[0136] In step S15, during the advancement of the endoscope, if the travel coordinates of the endoscope coincide with any positioning coordinate in the positioning coordinate set, an index relationship between the travel coordinates and the corresponding endoscope image is established.

[0137] This involves establishing an index relationship between travel coordinates and corresponding endoscopic images, enabling the search for the corresponding endoscopic image using the travel coordinates.

[0138] In step S16, the endoscope images acquired by the endoscope at the position where the travel coordinates and positioning coordinates coincide, along with the index relationship, are saved to the image database.

[0139] In this embodiment of the disclosure, the index relationship between the travel coordinates and the endoscope images is stored in the image database, as well as the endoscope images acquired at the locations where the travel coordinates and the positioning coordinates coincide. In this way, the endoscope images can be retrieved by inputting any coordinates based on the index relationship.

[0140] In one implementation, if the input coordinates are not coordinates in the positioning coordinate set, the advance distance between the input coordinates and each positioning coordinate in the positioning coordinate set corresponding to the endoscopic image can be calculated; wherein, the advance distance is the distance from the image coordinates to the positioning coordinates, or the advance distance is the distance from the positioning coordinates to the image coordinates; the positioning coordinate with the smallest advance distance in the positioning coordinate set is determined as the target image coordinate, and then, according to the index relationship, the endoscopic image corresponding to the target image coordinate is displayed as the target endoscopic image.

[0141] The above technical solution reduces the amount of data to be saved by dividing the planned advancement path into multiple planned advancement sub-paths and then acquiring and saving images at the positioning coordinates of the planned advancement sub-paths. By establishing an index relationship between the advancement coordinates and the corresponding endoscopic images, the corresponding endoscopic images can be quickly found based on the input coordinates during the image presentation stage.

[0142] In one implementation, see Figure 3 As shown, step S16, which involves saving the endoscopic images acquired by the endoscope along the travel coordinates and the index relationship to the image database, includes the following steps.

[0143] In step S161, if the travel coordinates coincide with any of the positioning coordinates in the set of travel coordinates, it is determined whether the endoscope captures multiple endoscope images at the position where the coordinates coincide.

[0144] In one embodiment of this disclosure, when the travel coordinates coincide with any of the positioning coordinates in the set of travel coordinates, the duration of the endoscope's stay at the coincident position is determined. If the stay duration exceeds a preset duration threshold, it is determined whether the endoscope has captured multiple endoscopic images at the coincident position. The preset duration threshold may be determined based on the endoscope's image capture frequency.

[0145] In another embodiment of this disclosure, if the travel coordinates coincide with any positioning coordinate in the set of positioning coordinates, the number of times the endoscope coincides with that positioning coordinate is determined. If the number of coincidences is greater than a preset overlap threshold, it is determined whether the endoscope captured multiple endoscopic images at the coordinate overlap position.

[0146] In step S162, if the endoscope captures multiple endoscope images at the coordinate overlap position, the valuable endoscope image is determined from the multiple endoscope images.

[0147] In this embodiment of the disclosure, if the endoscope captures multiple endoscope images at the coordinate overlap position, the multiple endoscope images whose acquisition time is closest to the current time are determined as valuable endoscope images; or, the shooting angle of the endoscope when capturing each endoscope image is determined, and the endoscope images with no overlapping shooting angles are taken as valuable endoscope images.

[0148] In one implementation, since the endoscope captures images at a certain shooting frequency, the endoscope may not capture an image at the exact moment when the traveling coordinates and the positioning coordinates coincide. Instead, it can capture an endoscopic image at the point where the spatial distance between the traveling coordinates and the positioning coordinates is minimized, and use this image as the valuable endoscopic image at the point where the coordinates coincide.

[0149] In step S163, the valuable endoscopic images in the endoscopic images and their corresponding index relationships are saved to the image database.

[0150] In this embodiment of the disclosure, see Figure 4 As shown, in step S162, if the endoscope captures multiple endoscope images at the coordinate overlap position, the step of determining the valuable endoscope image from the multiple endoscope images includes the following steps.

[0151] In step S1621, if the endoscope captures multiple endoscope images at the coordinate overlap position, then each endoscope image is divided into a grid to obtain the endoscope image corresponding to each grid.

[0152] For example, each endoscopic image can be divided into M×N grids.

[0153] In step S1622, the contrast index of each grid is determined based on the edge detection operator of the endoscopic image of each grid.

[0154] In this embodiment of the disclosure, the edge detection operator can be obtained by the Sobel detection algorithm or by edge detection operators of other algorithms. This disclosure does not impose any special restrictions on this.

[0155] Optionally, in step S1622, the step of determining the contrast index of each grid based on the edge detection operator of the endoscopic image of each grid includes:

[0156] Edge detection is performed on the endoscope image corresponding to each grid, and the absolute value of the edge detection operator for each pixel coordinate in each endoscope image is calculated.

[0157] The absolute values ​​of the edge detection operators for the pixel coordinates corresponding to each grid are summed to obtain the edge detection sum value for each grid.

[0158] The contrast index for each grid is determined based on the grid weight and edge detection value.

[0159] In this embodiment of the disclosure, the contrast index C(i,j) corresponding to each grid is determined by the following calculation formula:

[0160]

[0161] Where (i,j) are the pixel coordinates of the grid in the endoscopic image, and w(i,j) is the grid weight corresponding to the (i,j)th grid. The weights of all grids can be exactly the same, or the grid weight of the central region grid can be greater than that of the edge and corner grids. |S(i,j)| is the absolute value of the edge detection operator for the (i,j)th grid.

[0162] In step S1623, the quality index corresponding to each grid is determined based on the brightness of the endoscopic image of each grid.

[0163] Optionally, in step S1623, the step of determining the quality index corresponding to each grid based on the brightness of the endoscopic image of each grid includes:

[0164] Construct a brightness histogram for each grid based on the brightness of the endoscopic image corresponding to each grid.

[0165] In one embodiment, a brightness histogram for each grid is constructed using a 5-bin histogram based on the brightness of the endoscopic image corresponding to each grid.

[0166] Based on preset selection parameters, the target bin is determined from the bins of each brightness histogram.

[0167] Using the above embodiments for illustration, for example, if the number of bins in the brightness histogram is 5, and the preset selection parameter is to use the 3rd, 4th, and 5th bins as target bins, that is, the preset selection parameter can be that the number of target bins is 3 and the starting bin value of the target bins is 3, then the 3rd, 4th, and 5th bins can be selected as target bins from the brightness histogram of the 5 bins corresponding to each grid.

[0168] Determine the number of pixels in each bin within the target bin, and the number of pixels in each bin within the luminance histogram.

[0169] In this embodiment of the disclosure, the number of pixels contained in each bin is determined by the maximum prime number of the camera in the endoscope, and the number of pixels contained in each bin is the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction of the camera in that bin.

[0170] The quality index for each grid is determined based on the weight value of each bin in the brightness histogram, the number of pixels contained in each bin in the brightness histogram, and the number of pixels contained in each bin corresponding to each target bin.

[0171] In this embodiment of the disclosure, the quality index Q corresponding to each grid can be determined by the following formula:

[0172]

[0173] Where m and n are preset selection parameters, Hist(i) is the number of pixels contained in the i-th target bin, Hist(j) is the number of pixels contained in the j-th bin in the luminance histogram, N is the total number of bins in the luminance histogram, and ω(j) is the weight function of the j-th bin in the luminance histogram.

[0174] In step S1624, the overall quality of each endoscope image is determined based on the contrast index and quality index corresponding to each grid of each endoscope image.

[0175] Optionally, in step S1624, the step of determining the overall quality of each endoscopic image based on the contrast index and quality index corresponding to each grid of each endoscopic image includes:

[0176] The maximum contrast index corresponding to each endoscopic image is taken as the target contrast index, and the maximum quality index corresponding to each endoscopic image is taken as the target quality index.

[0177] The sum of the contrast index values ​​is obtained by multiplying the weighted contrast index by the contrast index of each grid in the endoscopic image, dividing by the quotient of the target contrast index, and then summing the results.

[0178] The sum of the quality index values ​​is obtained by multiplying the product of the index weight of the quality index and the quality index of each grid of the endoscopic image, then dividing by the quotient of the target quality index.

[0179] The sum of the contrast index and the quality index is used to determine the overall quality of the endoscopic image.

[0180] In this embodiment of the disclosure, the total quality Z of the endoscopic image can be determined by the following formula:

[0181]

[0182] Where i represents the i-th grid in the endoscopic image, i = 1, 2, ..., M × N; C max Q represents the target quality index of the endoscopic image, specifically the contrast index with the highest value. max The target quality index of the endoscopic image is the quality index with the highest value. 's' is an adjustment coefficient used to adjust the weights of the quality index and the contrast index.

[0183] In step S1625, a valuable endoscopic image is determined from multiple endoscopic images based on the overall quality.

[0184] In this embodiment of the disclosure, the number of valuable endoscopic images at each location coordinate can be specified in the control parameter set. For example, if the number of valuable endoscopic images at each location coordinate is specified as 5, then the valuable endoscopic image at the location coordinate will be determined from the 5 endoscopic images with the highest overall quality value.

[0185] Optionally, see Figure 5 As shown, in step S12, the step of determining the positioning coordinates of each planned advancement sub-path includes the following steps.

[0186] In step S121, for each planned advancement sub-path, the path coordinates of the planned advancement sub-path are determined.

[0187] The path coordinates referred to in this step are the coordinates of all points on the planned advancement sub-path. In other words, they are the coordinates of all points along the path of the planned advancement sub-path during the endoscopic advancement process.

[0188] In step S122, the path coordinates that coincide with the coordinates of the preset positioning point in the planned advancement sub-path are determined as the positioning coordinates of the planned advancement sub-path.

[0189] The preset positioning points include the midpoint, either of the two endpoints, or any of the inflection points during the advancement process.

[0190] In this embodiment of the disclosure, a planned advancement sub-path has one and only one positioning coordinate. Therefore, the preset positioning point includes the midpoint, either of the two endpoints, or any of the inflection points during the advancement process. For example, if the preset positioning point is designated as the midpoint, the coordinates in the planned advancement sub-path that coincide with the midpoint are used as the positioning coordinates. As another example, if the preset positioning point is designated as the endpoint to be reached first, the coordinates in the planned advancement sub-path that coincide with that endpoint are used as the positioning coordinates.

[0191] Based on the same concept, this disclosure also provides an endoscopic image display device, see [link to relevant documentation]. Figure 6 As shown, the endoscopic image display device 500 includes: a determination module 510, a query module 520, a fusion generation module 530, and a display module 560.

[0192] The determining module 510 is configured to, in response to the received specified image coordinates of an endoscope image at a specified location, determine the target image coordinates of the specified image coordinates from a set of positioning coordinates corresponding to the endoscope image;

[0193] The query module 520 is configured to query multiple target endoscope images corresponding to the target image coordinates from the image database based on the index relationship between the endoscope's advancement coordinates and the valuable endoscope images. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate.

[0194] The fusion generation module 530 is configured to perform image equalization and repair processing on each of the target endoscope images, and fuse them to generate an endoscope display image;

[0195] Display module 540 is configured to display the endoscope image in the viewport of the human-computer interaction interface.

[0196] Optionally, the fusion generation module 530 is configured as follows:

[0197] The target endoscope image is subjected to brightness and color equalization processing to obtain a first backup endoscope image;

[0198] Glare repair is performed on the first backup endoscope image to obtain the second backup endoscope image;

[0199] The second backup endoscope image is transformed from the Cartesian coordinate system to the polar coordinate system to obtain a polar coordinate endoscope image;

[0200] Image feature point detection is performed on the polar coordinate endoscope image to obtain endoscope image feature points;

[0201] Based on the RANSAC algorithm, normal and abnormal feature points in the endoscopic image feature points are determined;

[0202] The abnormal feature points are removed from the feature points of the endoscopic image to obtain an endoscopic display image that retains only the normal feature points.

[0203] Optionally, the fusion generation module 530 is configured as follows:

[0204] The region where the normal feature points are located in each of the endoscopic display images is taken as the endoscopic image region to be fused for the corresponding endoscopic display images;

[0205] Based on the Gaussian pyramid algorithm, convolution and downsampling are performed on each of the endoscopic image regions to be fused to obtain Gaussian pyramid endoscopic images with multiple different resolution scales corresponding to each of the endoscopic image regions to be fused.

[0206] Based on the Laplacian pyramid algorithm, the corresponding Laplacian endoscopic image is generated by upsampling from the bottom layer image of each Gaussian pyramid endoscopic image.

[0207] Calculate the average value of the pixels corresponding to each layer in each of the Laplacian endoscope images;

[0208] Image layers with the same average pixel value are used as a common endoscope image layer. The Laplacian endoscope images are then fused to obtain the endoscope display image.

[0209] Optionally, the endoscopic image display device 500 includes: a storage module configured to:

[0210] Each endoscope image corresponding to the advancement coordinate is divided into a grid to obtain the endoscope image corresponding to each grid.

[0211] The contrast index of each grid is determined based on the edge detection operator of the endoscopic image of each grid.

[0212] The quality index corresponding to each grid is determined based on the brightness of the endoscopic image of each grid.

[0213] The overall quality of each endoscope image is determined based on the contrast index and the quality index corresponding to each grid of each endoscope image;

[0214] Based on the overall quality, valuable endoscopic images are determined from the plurality of endoscopic images.

[0215] Optionally, the storage module is configured as follows:

[0216] Construct a brightness histogram for each grid based on the brightness of the endoscopic image corresponding to each grid.

[0217] Based on preset selection parameters, the target bin is determined from the bins of each brightness histogram;

[0218] Determine the number of pixels contained in each bin of the target bin, and the number of pixels contained in each bin of the brightness histogram;

[0219] The quality index corresponding to each grid is determined based on the weight value of each bin in the brightness histogram, the number of pixels contained in each bin in the brightness histogram, and the number of pixels contained in each bin corresponding to each target bin.

[0220] Optionally, the storage module is configured as follows:

[0221] Edge detection is performed on the endoscope image corresponding to each grid, and the absolute value of the edge detection operator for each pixel coordinate in each endoscope image is calculated;

[0222] The absolute values ​​of the edge detection operators corresponding to the pixel coordinates of each grid are summed to obtain the edge detection sum value of each grid.

[0223] The contrast index corresponding to each grid is determined based on the grid weight corresponding to each grid and the edge detection value.

[0224] Optionally, the storage module is configured as follows:

[0225] The maximum contrast index corresponding to each of the endoscopic images is taken as the target contrast index, and the maximum quality index corresponding to each of the endoscopic images is taken as the target quality index.

[0226] The sum of the contrast index values ​​is obtained by multiplying the exponential weight of the contrast index by the contrast index of each grid of the endoscopic image, dividing by the quotient of the target contrast index, and then summing the results.

[0227] The sum of the product of the index weight of the quality index and the quality index of each grid of the endoscopic image, divided by the quotient of the target quality index, is used to obtain the quality index sum value.

[0228] The sum of the contrast index and the quality index is used to determine the overall quality of the endoscopic image.

[0229] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0230] Those skilled in the art should understand that the device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and other division methods may exist in actual implementation. For instance, multiple modules may be combined or integrated into one module. Furthermore, the modules described as separate components may or may not be physically separated. Each module can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. When implemented in hardware, it can be implemented wholly or partially in the form of an integrated circuit or chip.

[0231] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a first processor, implements the steps of the endoscopic image display method according to any one of the first aspects.

[0232] This disclosure also provides an electronic device, including:

[0233] The first memory, on which the computer program is stored;

[0234] A second processor is configured to execute the computer program in the first memory to implement the steps of the endoscopic image display method according to any one of the first aspects.

[0235] Figure 7 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment. The electronic device 600 can be configured as an endoscopic image display device, such as... Figure 7 As shown, the electronic device 600 may include: a third processor 601 and a second memory 602. The electronic device 600 may also include one or more of the following: a multimedia component 603, an input / output (I / O) interface 604, and a communication component 605.

[0236] The third processor 601 controls the overall operation of the electronic device 600 to complete all or part of the steps in the aforementioned endoscopic image display method. The second memory 602 stores various types of data to support the operation of the electronic device 600. This data may include, for example, instructions for any application or method operating on the electronic device 600, and application-related data, such as images. The second memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 603 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the second memory 602 or transmitted via the communication component 605. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 604 provides an interface between the third processor 601 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons used to operate the saving and display of endoscopic images. The communication component 605 is used for wired or wireless communication between the electronic device 600 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 605 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0237] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the endoscopic image display method described above.

[0238] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the endoscopic image display method described above. For example, the computer-readable storage medium may be the second memory 602 including the program instructions, which may be executed by a third processor 601 of the electronic device 600 to complete the endoscopic image display method described above.

[0239] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0240] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0241] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for displaying endoscopic images, characterized in that, include: In response to receiving specified image coordinates of an endoscope image to be displayed at a specific location, the target image coordinates of the specified image coordinates are determined from the set of positioning coordinates corresponding to the endoscope image; Based on the index relationship between the endoscope's advancement coordinates and the valuable endoscope images, multiple target endoscope images corresponding to the target image coordinates are queried from the image database. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate. The target endoscopic images are subjected to image equalization and repair processing, and then fused to generate an endoscopic display image; The endoscopic image is displayed in the viewport of the human-computer interaction interface; The multiple valuable endoscopic images corresponding to each of the aforementioned advancement coordinates are determined in the following manner: Each endoscope image corresponding to the advancement coordinate is divided into a grid to obtain the endoscope image corresponding to each grid. The contrast index of each grid is determined based on the edge detection operator of the endoscopic image of each grid. The quality index corresponding to each grid is determined based on the brightness of the endoscopic image of each grid. The overall quality of each endoscope image is determined based on the contrast index and the quality index corresponding to each grid of each endoscope image; Based on the overall quality, valuable endoscopic images are determined from multiple endoscopic images; The step of determining the quality index corresponding to each grid based on the brightness of the endoscopic image of each grid includes: Construct a brightness histogram for each grid based on the brightness of the endoscopic image corresponding to each grid. Based on preset selection parameters, the target bin is determined from the bins of each brightness histogram; Determine the number of pixels contained in each bin of the target bin, and the number of pixels contained in each bin of the brightness histogram; The quality index corresponding to each grid is determined based on the weight value corresponding to each bin in the brightness histogram, the number of pixels contained in each bin in the brightness histogram, and the number of pixels contained in each bin corresponding to each target bin. The step of determining the contrast index of each grid based on the edge detection operator of the endoscopic image of each grid includes: Edge detection is performed on the endoscope image corresponding to each grid, and the absolute value of the edge detection operator for each pixel coordinate in each endoscope image is calculated; The absolute values ​​of the edge detection operators corresponding to the pixel coordinates of each grid are summed to obtain the edge detection sum value of each grid. The contrast index corresponding to each grid is determined based on the grid weight corresponding to each grid and the edge detection value.

2. The endoscopic image display method according to claim 1, characterized in that, The step of performing image equalization and repair processing on each of the target endoscopic images, and fusing them to generate an endoscopic display image, includes: The target endoscope image is subjected to brightness and color equalization processing to obtain a first backup endoscope image; Glare repair is performed on the first backup endoscope image to obtain the second backup endoscope image; The second backup endoscope image is transformed from the Cartesian coordinate system to the polar coordinate system to obtain a polar coordinate endoscope image; Image feature point detection is performed on the polar coordinate endoscope image to obtain endoscope image feature points; Based on the RANSAC algorithm, normal and abnormal feature points in the endoscopic image feature points are determined; The abnormal feature points are removed from the feature points of the endoscopic image to obtain an endoscopic display image that retains only the normal feature points.

3. The endoscopic image display method according to claim 2, characterized in that, The step of removing the abnormal feature points from the endoscopic image feature points to obtain an endoscopic display image that retains only the normal feature points includes: The region where the normal feature points are located in each of the endoscopic display images is taken as the endoscopic image region to be fused for the corresponding endoscopic display images; Based on the Gaussian pyramid algorithm, convolution and downsampling are performed on each of the endoscopic image regions to be fused to obtain Gaussian pyramid endoscopic images with multiple different resolution scales corresponding to each of the endoscopic image regions to be fused. Based on the Laplacian pyramid algorithm, the corresponding Laplacian endoscopic image is generated by upsampling from the bottom layer image of each Gaussian pyramid endoscopic image. Calculate the average value of the pixels corresponding to each layer in each of the Laplacian endoscope images; Image layers with the same average pixel value are used as a common endoscope image layer. The Laplacian endoscope images are then fused to obtain the endoscope display image.

4. The endoscopic image display method according to claim 1, characterized in that, The step of determining the overall quality of each endoscope image based on the contrast index and the quality index corresponding to each grid of each endoscope image includes: The maximum contrast index corresponding to each of the endoscopic images is taken as the target contrast index, and the maximum quality index corresponding to each of the endoscopic images is taken as the target quality index. The sum of the contrast index values ​​is obtained by multiplying the exponential weight of the contrast index by the contrast index of each grid of the endoscopic image, dividing by the quotient of the target contrast index, and then summing the results. The sum of the product of the index weight of the quality index and the quality index of each grid of the endoscopic image, divided by the quotient of the target quality index, is used to obtain the quality index sum value. The sum of the contrast index and the quality index is used to determine the overall quality of the endoscopic image.

5. An endoscopic image display device, characterized in that, include: The determination module is configured to, in response to receiving specified image coordinates of an endoscope image at a specified location, determine the target image coordinates of the specified image coordinates from a set of positioning coordinates corresponding to the endoscope image; The query module is configured to query multiple target endoscope images corresponding to the target image coordinates from the image database based on the index relationship between the endoscope's advancement coordinates and the valuable endoscope images. The image database stores multiple valuable endoscope images corresponding to each advancement coordinate. The fusion generation module is configured to perform image equalization and repair processing on each of the target endoscope images, and fuse them to generate an endoscope display image; The display module is configured to display the endoscope image in the viewport of the human-computer interaction interface; The storage module is configured to divide each of the endoscope images corresponding to the advancement coordinates into a grid, thereby obtaining an endoscope image corresponding to each grid. The contrast index of each grid is determined based on the edge detection operator of the endoscopic image of each grid. The quality index corresponding to each grid is determined based on the brightness of the endoscopic image of each grid. The overall quality of each endoscope image is determined based on the contrast index and the quality index corresponding to each grid of each endoscope image; Based on the overall quality, valuable endoscopic images are determined from multiple endoscopic images; The storage module is configured as follows: Construct a brightness histogram for each grid based on the brightness of the endoscopic image corresponding to each grid. Based on preset selection parameters, the target bin is determined from the bins of each brightness histogram; Determine the number of pixels contained in each bin of the target bin, and the number of pixels contained in each bin of the brightness histogram; The quality index corresponding to each grid is determined based on the weight value corresponding to each bin in the brightness histogram, the number of pixels contained in each bin in the brightness histogram, and the number of pixels contained in each bin corresponding to each target bin. The storage module is configured as follows: Edge detection is performed on the endoscope image corresponding to each grid, and the absolute value of the edge detection operator for each pixel coordinate in each endoscope image is calculated; The absolute values ​​of the edge detection operators corresponding to the pixel coordinates of each grid are summed to obtain the edge detection sum value of each grid. The contrast index corresponding to each grid is determined based on the grid weight corresponding to each grid and the edge detection value.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the first processor, the program implements the steps of the endoscopic image display method according to any one of claims 1-4.

7. An electronic device, characterized in that, include: The first memory, on which the computer program is stored; A second processor is configured to execute the computer program in the first memory to implement the steps of the endoscopic image display method according to any one of claims 1-4.

Images