Image processing method, system and device for eliminating laser delamination

Abstract

This application provides an image processing method, system, and device for eliminating laser delamination. This application locates and corrects the laser delamination area in an image frame so that the corrected laser delamination area and other image areas in the current image frame meet preset approximation conditions. This ensures that the display of the current image frame (including the corrected laser delamination area) is close to a normal image, avoiding the phenomenon of dynamic "bright light bars" appearing from time to time when it is displayed. This can reduce the visual discomfort of physicians during laser surgery and enable the surgery to achieve the desired effect smoothly.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, system and device for eliminating laser delamination. Background Art

[0002] Currently, during laser surgery, if the laser's release time (pulse width) within a frame's exposure is very short, typically hundreds of microseconds (µs), only a portion of the exposure lines will respond, while the remaining exposure lines will remain normally exposed. For example, during holmium laser surgery, if the laser's release time (pulse width) within a frame's exposure is very short, typically hundreds of µs (µs), only a portion of the exposure lines will respond, while the remaining exposure lines will remain normally exposed.

[0003] This situation will result in a layered image (one part includes the laser delamination area, the other part includes the normal image area). The laser delamination area is brighter or even overexposed, and will occasionally show dynamic "bright light bars" in subsequent image frames. This will affect the surgeon's operating field during laser surgery, interfere with the surgeon's judgment, and fail to achieve the desired surgical results. Summary of the Invention

[0004] The present application provides an image processing method for eliminating laser delamination, so as to eliminate laser delamination in an image frame.

[0005] The present invention provides an image processing method for eliminating laser delamination, the method comprising:

[0006] Obtaining a current image frame acquired in a laser application;

[0007] Locating a laser delamination area in the current image frame;

[0008] Determining a corresponding target correction scheme based on current designated image feature parameters of the laser delamination area;

[0009] The laser delamination area is corrected using the target correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet a preset approximation condition to eliminate the laser delamination in the current image frame.

[0010] The present invention provides an image processing device for eliminating laser delamination, the device comprising:

[0011] an obtaining unit, configured to obtain a current image frame collected in a laser application;

[0012] a positioning unit, configured to locate a laser delamination area in the current image frame;

[0013] a determination unit, configured to determine a corresponding target correction scheme based on current designated image feature parameters of the laser delamination area;

[0014] The correction unit is configured to correct the laser delamination area using the target correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet a preset approximation condition.

[0015] The embodiment of the present application further provides an image processing system for eliminating laser delamination, the system comprising: the device as described above, and a display terminal;

[0016] The display end is used to display the image frame; the original laser stratification area in the image frame meets the preset approximation condition with other image areas in the current image frame after correction, and the preset approximation condition is used to make the appearance of the original laser stratification area in the image frame after correction meet the approximate condition with the appearance of other image areas in the current image frame to eliminate the laser stratification in the current image frame.

[0017] An embodiment of the present application further provides an electronic device. The electronic device includes: a processor and a machine-readable storage medium;

[0018] The machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0019] The processor is used to execute machine-executable instructions to implement the steps of the above-disclosed method.

[0020] It can be seen from the above technical solution that this embodiment locates and corrects the laser stratification area existing in the image frame so that the corrected laser stratification area and other image areas in the current image frame meet the preset approximation conditions, thereby ensuring that the display of the current image frame (including the corrected laser stratification area) is close to the normal image, avoiding the phenomenon that it will occasionally present a dynamic "bright light bar" when being displayed, thereby reducing the visual discomfort of the physician during laser surgery and enabling the surgery to achieve the expected effect smoothly. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1a A schematic diagram of a layered image frame provided in an embodiment of the present application;

[0023] Figure 1b A schematic diagram of a normal image frame provided in an embodiment of the present application;

[0024] Figure 2 A flow chart of the method provided in the embodiment of the present application;

[0025] Figure 3 A schematic diagram of a correction scheme provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of boundary detection of a gradient-based digital image algorithm provided in an embodiment of the present application;

[0027] Figure 5 Schematic diagram of boundary detection using the line detection algorithm provided in an embodiment of the present application;

[0028] Figure 6 Schematic diagram of boundary detection under the edge extraction algorithm provided in an embodiment of the present application;

[0029] Figure 7 Schematic diagram of semantic segmentation model training and testing provided in an embodiment of the present application;

[0030] Figure 8 A structural diagram of the semantic segmentation model provided in an embodiment of the present application;

[0031] Figure 9 A diagram showing the training and testing structure of the target detection model provided in the embodiment of the present application;

[0032] Figure 10 A schematic diagram of the delamination effect provided by an embodiment of the present application;

[0033] Figure 11 A diagram of the device structure provided in an embodiment of the present application;

[0034] Figure 12 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present application.

[0036] The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit this application. The singular forms "a", "the" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates otherwise.

[0037] As described above, in laser surgery, if the release time (pulse width) of the laser within the exposure time of a frame of image is often very short (only a few hundred us), only a portion of the exposure rows will produce a "response", and the remaining exposure rows will still be normally exposed, resulting in stratification in the final frame of image (referred to as laser stratification). This laser stratification will cause a frame of image to be divided into two areas: the laser stratification area and the normal image area. The area size and position of the laser stratification area are not fixed, but the brightness of the laser stratification area is high or even overexposed. Taking holmium laser surgery as an example, stratification will occur in a frame of image collected during holmium laser surgery (referred to as holmium laser stratification). This holmium laser stratification will cause a frame of image to be divided into two areas: the holmium laser stratification area and the normal image area. Figure 1a An example of an image showing a region where holmium laser delamination is present. Figure 1b An image with normal exposure is shown as an example.

[0038] For the above-mentioned images with laser delamination, such as holmium laser delamination, dynamic "bright light bars" will appear from time to time when displayed, which will cause visual discomfort to doctors performing laser surgery, such as holmium laser surgery, and affect the operating field of view, interfere with the doctor's judgment, and lead to failure to achieve the expected surgical effect. To solve this technical problem, this embodiment provides an image processing method for eliminating laser delamination, such as holmium laser delamination, which can eliminate laser delamination in image frames. The following example describes:

[0039] See also Figure 2 , Figure 2 This is a flow chart of a method provided in an embodiment of the present application. This method can be applied to electronic devices. As an example, the electronic device can be a front-end device such as a camera in an endoscopic imaging system, or a medical device, a back-end server, etc., which is not specifically limited in this embodiment.

[0040] like Figure 2 As shown, the process may include the following steps:

[0041] Step 201: Obtain a current image frame captured in a laser application.

[0042] There are many ways to implement this step 201 in specific implementations, for example, obtaining the current image frame in the video stream collected by the endoscopic imaging system during laser surgery. Here, the current image frame can be any image frame in the video stream, and this embodiment does not specifically limit it. When collecting the current image frame, because the release time (pulse width) of the laser during the exposure time is very short (only a few hundred us), only a part of the exposure rows will produce a "response", and the remaining exposure rows will still be normally exposed, resulting in stratification (referred to as laser stratification) in the final current image frame. This laser stratification will cause a frame of image to be divided into two areas: a laser stratification area and a normal image area. That is, the current image frame contains two areas: a laser stratification area and a normal image area. Applied to holmium laser surgery, the laser stratification area contained in the current image frame may be a holmium laser stratification area.

[0043] Step 202: locate the laser delamination area in the current image frame.

[0044] As an embodiment, this step 202 can detect and locate the laser stratification area in the current image frame based on an image processing algorithm (an algorithm for detection and positioning), such as the above-mentioned holmium laser stratification area. An example will be given below and will not be repeated here.

[0045] As another embodiment, this step 202 can also detect and locate the laser stratification area in the current image frame based on the trained deep learning model, such as the above-mentioned holmium laser stratification area. An example will be given below and will not be repeated here.

[0046] Step 203: Determine a corresponding target correction solution based on the current designated image feature parameters of the laser delamination area.

[0047] In this embodiment, there are many ways to implement the target correction scheme based on the current specified image characteristic parameters of the laser stratification area, such as the above-mentioned holmium laser stratification area. For example, through experiments, different characteristic parameter intervals are set for the specified image characteristic parameters, and different characteristic parameter intervals correspond to different correction schemes. Under this premise, the characteristic parameter interval range in which the current specified image characteristic parameter is located can be found from the preset characteristic parameter interval ranges, and the correction scheme configured for the characteristic parameter interval range is determined as the target correction scheme. In a specific implementation, the above-mentioned specified image characteristic parameter is, for example, brightness, and the corresponding characteristic parameter interval range can be characterized by a brightness threshold.

[0048] Based on the above description, this embodiment can adopt multiple dimensional correction schemes to correct and repair laser stratification of different severity, which can not only meet the different needs of different doctors, but also promptly respond to and solve laser stratification in different situations. Optionally, in this embodiment, 9 characteristic parameter intervals can be generally divided, and correspondingly, there can be 9 correction schemes. Figure 3 The nine correction schemes are shown as examples, and the correction schemes will be described in detail below, which will not be repeated here. In this embodiment, the target correction scheme may include the at least one correction scheme mentioned above.

[0049] Step 204 : Correct the laser delamination area using a target correction solution so that the corrected laser delamination area and other image areas in the current image frame meet a preset approximation condition to eliminate the laser delamination in the current image frame.

[0050] In this embodiment, the preset approximation condition can be set based on actual circumstances. It is used to ensure that the appearance of the original laser delamination area in the image frame after correction is similar (e.g., nearly identical) to the appearance of other image areas in the current image frame, thereby eliminating the laser delamination in the current image frame. For example, the brightness of the corrected laser delamination area and the brightness of other image areas in the current image frame can meet the brightness approximation condition, and the difference in the overall brightness average between the two is less than or equal to a set threshold, etc., which is not specifically limited in this embodiment. The following will describe how to implement step 204 based on a specific target correction solution, and will not be elaborated here.

[0051] So far, completed Figure 2 The process shown.

[0052] pass Figure 2 As can be seen from the process shown, this embodiment locates and corrects the laser stratification area existing in the image frame so that the corrected laser stratification area, such as the holmium laser stratification area, and other image areas in the current image frame meet the preset approximation conditions, thereby ensuring that the display of the current image frame (including the corrected laser stratification area) is close to the normal image, avoiding the phenomenon that it will occasionally show dynamic "bright light bars" when being displayed, thereby reducing the visual discomfort of the physician during laser surgery and enabling the surgery to achieve the expected effect smoothly.

[0053] It should be noted that, in this embodiment, the above Figure 2 The process shown can be realized in automatic mode. Once the automatic mode is turned on, the above Figure 2 The process shown in the figure performs real-time processing on the image frames in the captured video stream. Figure 2 The steps in the process shown are used to eliminate the laser delamination area in the image frame. As another embodiment, the above Figure 2The illustrated process can also be implemented through external triggering, such as physician-triggered instructions (the device carries the laser delamination area removal degree level to adapt to different situations), to support real-time switching between different removal degree levels during surgery. Here, the laser delamination area removal degree level can correspond to the above-mentioned correction scheme, with different levels corresponding to different correction schemes.

[0054] The following describes how to locate the laser delamination area in the current image frame based on the above-mentioned image processing algorithm for detection and positioning:

[0055] Gradient-based digital image algorithms:

[0056] In an image frame, a laser delamination region, such as a holmium laser delamination region, has a relatively clear horizontal boundary from other image regions in the image frame. Based on this, in this embodiment, a gradient-based digital image algorithm can be used to locate the laser delamination region, such as the holmium laser delamination region, in the current image frame.

[0057] As an embodiment, the horizontal gradient and vertical gradient of each pixel point in the current image frame can be calculated based on a gradient-based digital image algorithm, and then the boundary of the laser stratification area can be determined based on the distribution of the horizontal gradient and vertical gradient of the pixel points in each pixel row in the current image frame. Figure 4 An example of boundary detection using a gradient-based digital image algorithm is shown.

[0058] Optionally, in this embodiment, there are many ways to calculate the horizontal gradient of a pixel. For example, for each pixel, if there is a neighboring pixel in a first specified direction, the horizontal gradient of the pixel is calculated using the grayscale value of the pixel and the grayscale value of the neighboring pixel. Otherwise, the horizontal gradient of the pixel is calculated using the grayscale value of the pixel and the specified grayscale value. Here, the first specified direction, such as the right side, is not specifically limited in this embodiment.

[0059] Optionally, in this embodiment, there are many ways to calculate the vertical gradient of a pixel. For example, for each pixel, if there is a neighboring pixel in a second specified direction, the vertical gradient of the pixel is calculated using the grayscale value of the pixel and the grayscale value of the neighboring pixel. Otherwise, the vertical gradient of the pixel is calculated using the grayscale value of the pixel and the specified grayscale value. Here, the second specified direction, such as downward, is not specifically limited in this embodiment.

[0060] In this embodiment, in the pixel row at the boundary of a laser delamination region, such as a holmium laser delamination region, the horizontal gradient span of the pixel point is relatively small (generally less than a set horizontal gradient threshold), but the vertical gradient span is relatively large (generally greater than the set vertical gradient threshold). Based on this, this embodiment can determine the relatively small horizontal gradient span (generally less than the set horizontal gradient threshold) and the relatively large vertical gradient span (generally greater than the set vertical gradient threshold) of the pixel point as the first specified boundary condition. Based on this, in this embodiment, the pixel row whose horizontal and vertical gradient distributions meet the first specified boundary condition can be determined as the laser delamination region boundary. The corresponding laser delamination region can then be determined based on the laser delamination region boundary. For example, if the upper and lower boundaries of the laser delamination region are determined, the area between the upper and lower boundaries in the current image frame can be used as the laser delamination region.

[0061] Straight line detection method:

[0062] In this embodiment, a line detection method such as the Hough transform, LSD, or CannyLines can be used to locate a laser delamination region, such as a holmium laser delamination region, in the current image frame. For example, a line detection method such as the Hough transform, LSD, or CannyLines can be used to detect all possible lines within the current image frame (the detected lines do not include the boundary of the current image frame). Then, based on the length and inclination of each line, two lines whose length and inclination meet the second specified boundary condition are determined as the laser delamination region boundary. Figure 5 An example of boundary detection under the line detection algorithm is shown.

[0063] In this embodiment, the boundary of the laser delamination area, such as the holmium laser delamination area, can be determined through experimental testing. Generally, it is the two longest straight lines in the image frame. Among them, the length of the two straight lines is greater than the set length threshold and the inclination is less than the set angle threshold (generally the inclination is very small). Based on this, the length of the two straight lines greater than the set length threshold and the inclination less than the set angle threshold (generally the inclination is very small) can be used as the above-mentioned second specified boundary condition. Based on this, in this embodiment, the two straight lines whose length and inclination meet the second specified boundary condition can be determined as the laser delamination area boundary. Thereafter, the corresponding laser delamination area can be determined based on the laser delamination area boundary. For example, if the upper boundary and the lower boundary of the laser delamination area are determined, the area between the upper boundary and the lower boundary in the current image frame can be used as the laser delamination area, such as the holmium laser delamination area.

[0064] Edge extraction algorithm:

[0065] In this embodiment, an edge extraction algorithm can be used to locate a laser delamination area, such as a holmium laser delamination area, in the current image frame. For example, the current image frame is first converted into a grayscale image, and then the edge detection of the grayscale image is performed using a set edge extraction operator to obtain an edge sub-image, and the edge sub-image is adaptively thresholded to obtain a binary image; the binary image is smoothed using a preset morphological processing method and useless areas that do not meet the set delamination area, such as short lines, are removed to obtain at least one area to be processed; and a laser delamination area, such as a holmium laser delamination area, is determined based on the obtained area to be processed. Optionally, in this embodiment, there are many ways to implement the determination of the laser delamination area based on the obtained area to be processed, such as extracting at least one maximum connected domain from the obtained area to be processed, determining the wide center line of the circumscribed rectangle of the extracted maximum connected domain as the boundary of the laser delamination area, and determining the laser delamination area based on the boundary of the laser delamination area. Figure 6 An example of boundary detection under the edge extraction algorithm is shown.

[0066] The above example describes how to use an algorithm to locate the laser delamination area in the current image frame. The following describes how to locate the holmium laser delamination area in the current image frame based on a trained deep learning model for detection and positioning:

[0067] Semantic Segmentation Model:

[0068] In this embodiment, a semantic segmentation model can be trained based on the training images and laser layer labels in the training images, such as holmium laser layer labels, loss functions, and network structures. Of course, after the semantic segmentation model is trained, the semantic segmentation model can also be tested using test images, as shown in the following example. Figure 7 As shown. The final trained deep learning semantic segmentation model mainly consists of an encoding network and a decoding network. The structure example is as follows Figure 8 As shown in Figure 2, the encoding network usually consists of convolution and downsampling, and the decoding network usually consists of convolution and upsampling, such as UNet.

[0069] Based on the above-mentioned semantic segmentation model, in this embodiment, the current image frame can be input into the semantic segmentation model, so that the semantic segmentation model can identify the laser layered area blocks in the current image frame, such as the holmium laser layered area blocks, based on the semantic features of the current image frame and output them.

[0070] Of course, in the specific implementation, the output result of the semantic segmentation model may be that there are more than two scattered laser stratification area blocks in the current image frame, such as holmium laser stratification area blocks. Under this premise, the output result can be subjected to morphological and interference removal to accurately locate the laser stratification area blocks in the current image frame, such as holmium laser stratification area blocks.

[0071] Object Detection Model:

[0072] In this embodiment, the training and testing of the target detection model are similar to the semantic segmentation model described above, except that compared to the semantic segmentation model, the target detection model can directly frame the target area, i.e., the laser stratification area block, such as the specific location of the holmium laser stratification area (which can be represented by the four vertices or center points of the laser stratification area). The deep learning target detection model is mainly composed of a feature extraction network and a detection head network, such as Figure 9 The feature extraction network is a convolutional neural network such as VGG, ResNet, and EfficientNet. The detection head network consists of multiple convolutional layers and performs classification and regression calculations to identify and locate laser delamination areas, such as the holmium laser delamination area.

[0073] Based on the above-mentioned target detection model, in this embodiment, the current image frame is input into the trained target detection model, so that the target detection model can identify the position information of the laser stratification area in the current image frame, such as the holmium laser stratification area, based on the depth features of the current image frame and output it.

[0074] The above describes an example of how to locate the laser delamination area in the current image frame based on the trained deep learning model for detection and positioning.

[0075] The following describes how to use the target correction scheme in step 204 to correct the laser delamination area in the current image frame:

[0076] In this step 204, the target correction scheme is used to correct the laser stratification area in the current image frame. The purpose is to make the corrected laser stratification area, such as the holmium laser stratification area, visually close to other image areas in the current image frame (reflected by the corrected laser stratification area and other image areas in the current image frame meeting preset approximation conditions) to eliminate the laser stratification in the current image frame.

[0077] Based on Figure 3 For example, in step 104, the target correction scheme may include Figure 3 At least one correction scheme is shown to achieve different degrees of elimination of laser delamination, such as holmium laser delamination, in the current image frame.

[0078] As an embodiment, when the target correction scheme includes at least a brightness correction scheme, the brightness of the laser stratification area, such as the holmium laser stratification area, in the current image frame can be corrected based on the brightness correction scheme. Here, the brightness correction scheme is used to adjust the brightness of the laser stratification area, such as the holmium laser stratification area, in the current image frame. For the laser stratification area, such as the holmium laser stratification area, in the current image frame caused by exposure abnormality, the brightness correction scheme can be used to make the brightness value of the corrected laser stratification area, such as the holmium laser stratification area, and the brightness values ​​of other image areas meet the preset brightness approximation conditions (such as the difference in brightness value is within the preset brightness threshold range), and finally the brightness of the corrected laser stratification area, such as the holmium laser stratification area, can be adjusted to be basically consistent with the brightness of other image areas in the current image frame. Here, the above-mentioned preset brightness threshold range is set based on the purpose of basically consistent brightness.

[0079] Optionally, in this embodiment, the above-mentioned brightness correction scheme can be specifically implemented as follows: using color spaces including but not limited to YUV, HSV, and LAB brightness spaces, and adjusting the brightness of the laser stratification area in the current image frame, such as the holmium laser stratification area, through traditional image processing methods such as gamma correction, logarithmic transformation, and histogram statistical mapping.

[0080] As an embodiment, when the target correction scheme includes at least a color correction scheme, the color of the laser stratification area, such as the holmium laser stratification area, in the current image frame can be corrected based on the color correction scheme. Here, the color correction scheme is used to adjust the color of the laser stratification area, such as the holmium laser stratification area, in the current image frame, so that the overall hue of the corrected laser stratification area and the hue of other image areas in the current image frame meet a preset color approximation condition (for example, the difference between the color value of the corrected laser stratification area and the color value of other image areas in the current image frame is within a preset color threshold range), and finally adjust the overall hue of the laser stratification area, such as the holmium laser stratification area, in the current image frame to an effect that is basically consistent with other areas of the image. Here, the above-mentioned preset color threshold range is set based on the purpose of basically consistent overall hue.

[0081] Optionally, in this embodiment, when the color correction scheme is specifically implemented, traditional tone mapping algorithms such as histogram normalization, Reinhard, and fuzzy clustering can be used to migrate the color features of other image areas in the current image frame to the laser layered area in the current image frame, such as the holmium laser layered area, so as to achieve the purpose of adjusting the overall color tone of the laser layered area in the current image frame, such as the holmium laser layered area, to be basically consistent with other areas of the image.

[0082] As an embodiment, when the target correction scheme includes at least a boundary correction scheme, the boundary of the laser stratification area in the current image frame, such as the holmium laser stratification area, can be corrected based on the boundary correction scheme. For example, the boundary of the laser stratification area in the current image frame, such as the holmium laser stratification area, can be smoothed so that the corrected laser stratification area and other image areas in the current image frame meet the preset fusion conditions. Finally, the horizontal boundary of the laser stratification area with obvious stratification in the current image frame, such as the holmium laser stratification area, can be smoothly transitioned to better fuse with the image of the surrounding area, so that the feature distribution of the laser stratification area in the current image frame, such as the holmium laser stratification area, can be more closely matched with the other surrounding image areas.

[0083] Optionally, the boundary correction scheme can be implemented by: utilizing traditional image restoration algorithms such as FMM, Criminisi, TV model, BSBC model, PatchMatch, or directly replacing the laser stratification area in the current image frame, such as the holmium laser stratification area, with other surrounding areas and then performing weighted fusion processing to better repair the horizontal boundary of the laser stratification area in the current image frame, such as the holmium laser stratification area.

[0084] As an embodiment, when the target correction scheme includes at least adaptive correction, the laser stratification area, such as the holmium laser stratification area, in the current image frame can be corrected based on the adaptive correction scheme. Here, the adaptive correction scheme is implemented using a deep learning network model to achieve end-to-end adaptive correction of the laser stratification area, such as the holmium laser stratification area, in the current image frame. The deep learning network model can be a generative adversarial network, which is trained by learning the depth features of the surrounding image of the laser stratification, such as the holmium laser stratification area. The deep learning network model can obtain information such as the brightness and color of the laser stratification area, such as the holmium laser stratification area, in the current image frame, and then the obtained information is directly adjusted in one step to an effect consistent with other image areas in the current image frame, such as basically consistent brightness, color, etc. Here, for basically consistent brightness, please refer to the description of the brightness correction scheme above, and for basically consistent color, please refer to the description of the color correction scheme above.

[0085] As an embodiment, when the target correction scheme includes at least a previous frame reference correction scheme, the laser stratification area in the current image frame, such as the holmium laser stratification area, can be corrected based on the previous frame reference correction scheme. For example, the previous frame image of the current image frame in the video stream is intercepted, and the area at the same position as the laser stratification area, such as the holmium laser stratification area, in the current image frame is obtained from the previous frame image (recorded as the reference area), and then the reference area is effectively fused with the laser stratification area, such as the holmium laser stratification area, in the current image frame, thereby eliminating the laser stratification area, such as the holmium laser stratification area, in the current image frame.

[0086] The above describes the correction of the laser delamination area using the target correction scheme.

[0087] Optionally, in this embodiment, after correcting the laser delamination area in the current image frame using the target correction solution, the corrected current image frame can be displayed, where the laser delamination, such as holmium laser delamination, is eliminated from the current image frame.

[0088] Optionally, in this embodiment, the corrected current image frame may be displayed on a display screen for the physician to watch on the screen. Figure 10 The effects of eliminating laser delamination in the current image frame by using the above different correction schemes are shown as examples. Figure 10 It can be seen that different correction schemes (corresponding elimination degrees) produce different processing effects, but are not limited to these effects.

[0089] The above describes the method provided in the embodiment of the present application. The following describes the device provided in the embodiment of the present application:

[0090] See also Figure 11 , Figure 11 This is a diagram of the device structure provided in the embodiment of the present application. The device includes:

[0091] an obtaining unit, configured to obtain a current image frame collected in a laser application;

[0092] a positioning unit, configured to locate a laser delamination area in the current image frame;

[0093] a determination unit, configured to determine a corresponding target correction scheme based on current designated image feature parameters of the laser delamination area;

[0094] The correction unit is configured to correct the laser delamination area using the target correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet a preset approximation condition.

[0095] Optionally, in this embodiment, the current image frame is a current image frame in a video stream captured by an endoscopic imaging system during laser surgery.

[0096] Optionally, in this embodiment, locating the laser stratification area in the current image frame includes: locating the laser stratification area in the current image frame based on an image processing algorithm for detection and positioning; or locating the laser stratification area in the current image frame based on a trained deep learning model for detection and positioning.

[0097] Optionally, in this embodiment, if the image processing algorithm is a gradient-based digital image algorithm, the image processing algorithm based on detection and positioning, locating the laser stratification area in the current image frame includes: calculating the horizontal gradient and vertical gradient of the pixel points in the current image frame, determining the boundary of the laser stratification area based on the distribution of the horizontal gradient and vertical gradient of the pixel points in each pixel row in the current image frame, and determining the laser stratification area based on the laser stratification area boundary; wherein, the distribution of the horizontal gradient and vertical gradient of the pixel points in the pixel row corresponding to the laser stratification area boundary meets the first specified boundary condition.

[0098] Optionally, in this embodiment, if the image processing algorithm is a straight line detection method, the image processing algorithm for detection and positioning based on which the laser stratification area is located in the current image frame includes: using the straight line detection method to detect straight lines in the current image frame, the detected straight lines do not contain the boundaries of the current image frame; based on the length and inclination of each straight line, two straight lines whose length and inclination meet the second specified boundary condition are determined as the boundaries of the laser stratification area, and the laser stratification area is determined based on the boundaries of the laser stratification area.

[0099] Optionally, in this embodiment, if the image processing algorithm is an edge extraction algorithm, the image processing algorithm based on detection and positioning, locating the laser stratification area in the current image frame includes: converting the current image frame into a grayscale image; performing edge detection on the grayscale image using a set edge extraction operator to obtain an edge sub-image, and performing adaptive thresholding on the edge sub-image to obtain a binarized image; using a preset morphological processing method to smooth the binarized image and remove useless areas that do not meet the set stratification area to obtain at least one area to be processed; and determining the laser stratification area based on the obtained area to be processed.

[0100] Optionally, in this embodiment, determining the laser delamination area based on the obtained area to be processed includes: extracting at least one maximum connected domain from the obtained area to be processed, determining the wide center line of the circumscribed rectangle of the extracted maximum connected domain as the boundary of the laser delamination area, and determining the laser delamination area based on the boundary of the laser delamination area.

[0101] Optionally, in this embodiment, if the deep learning model is a trained semantic segmentation model, locating the laser stratification area in the current image frame based on the trained deep learning model for detection and positioning includes: inputting the current image frame into the trained semantic segmentation model, so that the semantic segmentation model identifies and outputs the laser stratification area block in the current image frame based on the semantic features of the current image frame.

[0102] Optionally, in this embodiment, if the deep learning model is a trained target detection model, locating the laser stratification area in the current image frame based on the trained deep learning model for detection and positioning includes: inputting the current image frame into the trained target detection model, so that the target detection model identifies and outputs the position information of the laser stratification area in the current image frame based on the depth features of the current image frame.

[0103] Optionally, in this embodiment, based on the current specified image characteristic parameters of the laser stratification area, determining the corresponding target correction scheme includes: searching the characteristic parameter interval range in which the specified image characteristic parameters are located from the preset characteristic parameter interval ranges; and determining the configured correction scheme for the characteristic parameter interval range as the target correction scheme.

[0104] Optionally, in this embodiment, if the target correction scheme includes at least a brightness correction scheme, the correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximate conditions includes: correcting the brightness of the laser delamination area using the brightness correction scheme so that the brightness of the corrected laser delamination area and the brightness of other image areas in the current image frame meet preset brightness approximate conditions; and / or,

[0105] If the target correction scheme includes at least a color correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximation conditions includes: correcting the color of the laser delamination area using the color correction scheme so that the overall hue of the corrected laser delamination area and the hue of other image areas in the current image frame meet preset color approximation conditions; and / or

[0106] If the target correction scheme includes at least a boundary correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximation conditions includes: smoothing the boundary of the laser delamination area using the boundary correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet preset fusion conditions; and / or

[0107] If the target correction scheme includes at least an adaptive correction scheme, correcting the laser delamination area using the target correction scheme includes: correcting the laser delamination area using a trained deep learning network model for adaptive correction; the deep learning network model for adaptive correction is obtained by learning depth features of images around the laser delamination area; and / or,

[0108] If the target correction scheme includes at least a previous frame reference correction scheme, the use of the target correction scheme to correct the laser stratification area so that the image parameters of the corrected laser stratification area and the image parameters of other image areas in the current image frame meet preset approximation conditions includes: obtaining a reference area at the same position as the laser stratification area from the previous frame image of the current image frame, and effectively fusing the reference area with the laser stratification area in the current image frame.

[0109] So far, completed Figure 11 The device structure diagram shown.

[0110] See also Figure 12 , Figure 12 This is a system structure diagram provided in the embodiment of this application. The system includes: Figure 11 The device and display shown;

[0111] Here, the display end is used to display the image frame; the original laser stratification area in the image frame meets the preset approximation condition with other image areas in the current image frame after correction, and the preset approximation condition is used to make the appearance of the original laser stratification area in the image frame after correction meet the approximate condition with the appearance of other image areas in the current image frame to eliminate the laser stratification in the current image frame.

[0112] The present application also provides Figure 11 The hardware structure of the device shown. Figure 12 , Figure 12 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 12 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.

[0113] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0114] Exemplarily, the machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0115] The systems, devices, modules or units described in the above embodiments may be implemented by a computer processor or entity, or by a product with certain functions.

[0116] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0117] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] The present application is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0119] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0121] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method for eliminating laser delamination, characterized in that: The method includes: obtaining a current image frame from a holmium laser video stream acquired by an endoscopic imaging system during holmium laser surgery; Locating a laser delamination region in the current image frame; the laser delamination region is generated by the interaction between the holmium laser and the tissue; the current image frame containing the laser delamination region presents a dynamic bright light bar when displayed; Determining a corresponding target correction scheme based on the current designated image characteristic parameters of the laser delamination area; if the designated image characteristic parameters are within different characteristic parameter intervals, the corresponding correction schemes will be different; The laser delamination area is corrected using the target correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet a preset approximation condition to eliminate the laser delamination in the current image frame.

2. The method according to claim 1, characterized in that The locating of the laser delamination area in the current image frame includes: Locating the laser delamination area in the current image frame based on an image processing algorithm for detection and positioning; or Based on the trained deep learning model for detection and positioning, the laser delamination area is located in the current image frame.

3. The method according to claim 2, characterized in that If the image processing algorithm is a gradient-based digital image algorithm, locating the laser delamination area in the current image frame based on the image processing algorithm for detection and positioning includes: calculating horizontal gradients and vertical gradients of pixels in the current image frame, determining a boundary of the laser delamination area based on the distribution of the horizontal gradients and vertical gradients of pixels in each pixel row in the current image frame, and determining the laser delamination area based on the laser delamination area boundary; wherein the distribution of the horizontal gradients and vertical gradients of pixels in the pixel row corresponding to the laser delamination area boundary satisfies a first specified boundary condition; If the image processing algorithm is a line detection method, locating the laser delamination area in the current image frame based on the image processing algorithm for detection and positioning includes: detecting straight lines in the current image frame using the line detection method, wherein the detected straight lines do not include a boundary of the current image frame; determining, based on the length and inclination of each straight line, two straight lines whose lengths and inclinations satisfy a second specified boundary condition as boundaries of the laser delamination area, and determining the laser delamination area based on the boundaries of the laser delamination area; If the image processing algorithm is an edge extraction algorithm, the image processing algorithm based on detection and positioning, locating the laser stratification area in the current image frame includes: converting the current image frame into a grayscale image; using a set edge extraction operator to perform edge detection on the grayscale image to obtain an edge sub-image, and performing adaptive thresholding processing on the edge sub-image to obtain a binarized image; using a preset morphological processing method to smooth the binarized image and remove useless areas that do not meet the set stratification area to obtain at least one area to be processed; and determining the laser stratification area based on the obtained area to be processed.

4. The method according to claim 3, characterized in that Determining the laser delamination area based on the obtained area to be processed includes: At least one maximum connected domain is extracted from the obtained area to be processed, a wide center line of a circumscribed rectangle of the extracted maximum connected domain is determined as a boundary of a laser delamination area, and the laser delamination area is determined according to the boundary of the laser delamination area.

5. The method according to claim 2, characterized in that If the deep learning model is a trained semantic segmentation model, locating the laser delamination area in the current image frame based on the trained deep learning model for detection and positioning includes: inputting the current image frame into the trained semantic segmentation model, so that the semantic segmentation model identifies and outputs the laser delamination area block in the current image frame based on semantic features of the current image frame; If the deep learning model is a trained target detection model, locating the laser stratification area in the current image frame based on the trained deep learning model for detection and positioning includes: inputting the current image frame into the trained target detection model, so that the target detection model identifies and outputs the position information of the laser stratification area in the current image frame based on the depth features of the current image frame.

6. The method according to claim 1, characterized in that Determining a corresponding target correction scheme based on the current specified image feature parameters of the laser delamination area includes: Finding the feature parameter interval range where the specified image feature parameter is located from the preset feature parameter interval ranges; The configured correction scheme for the characteristic parameter interval range is determined as the target correction scheme.

7. The method according to claim 1 or 6, characterized in that If the target correction scheme includes at least a brightness correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximate conditions includes: correcting the brightness of the laser delamination area using the brightness correction scheme so that the brightness of the corrected laser delamination area and the brightness of other image areas in the current image frame meet preset brightness approximate conditions; and / or, If the target correction scheme includes at least a color correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximation conditions includes: correcting the color of the laser delamination area using the color correction scheme so that the overall hue of the corrected laser delamination area and the hue of other image areas in the current image frame meet preset color approximation conditions; and / or, If the target correction scheme includes at least a boundary correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximation conditions includes: smoothing the boundary of the laser delamination area using the boundary correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet preset fusion conditions; and / or If the target correction scheme includes at least an adaptive correction scheme, correcting the laser delamination area using the target correction scheme includes: correcting the laser delamination area using a trained deep learning network model for adaptive correction; the deep learning network model for adaptive correction is obtained by learning depth features of images around the laser delamination area; and / or, If the target correction scheme includes at least a previous frame reference correction scheme, the use of the target correction scheme to correct the laser stratification area so that the image parameters of the corrected laser stratification area and the image parameters of other image areas in the current image frame meet preset approximation conditions includes: obtaining a reference area at the same position as the laser stratification area from the previous frame image of the current image frame, and effectively fusing the reference area with the laser stratification area in the current image frame.

8. An image processing device for eliminating laser delamination, characterized in that: The device includes: an acquisition unit, configured to acquire a current image frame from a holmium laser video stream acquired by an endoscopic imaging system during holmium laser surgery; a positioning unit for locating a laser delamination region in the current image frame; the laser delamination region is generated by the interaction between the holmium laser and the tissue; the current image frame containing the laser delamination region presents a dynamic bright light bar when displayed; A determination unit, configured to determine a corresponding target correction scheme based on the current designated image characteristic parameters of the laser delamination area; different correction schemes correspond to different characteristic parameter intervals within which the designated image characteristic parameters fall; The correction unit is configured to correct the laser delamination area using the target correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet a preset approximation condition.

9. The device according to claim 8, characterized in that The locating of the laser delamination area in the current image frame includes: locating the laser delamination area in the current image frame based on an image processing algorithm for detection and positioning; or locating the laser delamination area in the current image frame based on a trained deep learning model for detection and positioning; If the image processing algorithm is a gradient-based digital image algorithm, locating the laser delamination area in the current image frame based on the image processing algorithm for detection and positioning includes: calculating horizontal gradients and vertical gradients of pixels in the current image frame, determining a boundary of the laser delamination area based on the distribution of the horizontal gradients and vertical gradients of pixels in each pixel row in the current image frame, and determining the laser delamination area based on the laser delamination area boundary; wherein the distribution of the horizontal gradients and vertical gradients of pixels in the pixel row corresponding to the laser delamination area boundary satisfies a first specified boundary condition; If the image processing algorithm is a line detection method, locating the laser delamination area in the current image frame based on the image processing algorithm for detection and positioning includes: detecting straight lines in the current image frame using the line detection method, wherein the detected straight lines do not include a boundary of the current image frame; determining, based on the length and inclination of each straight line, two straight lines whose lengths and inclinations satisfy a second specified boundary condition as boundaries of the laser delamination area, and determining the laser delamination area based on the boundaries of the laser delamination area; If the image processing algorithm is an edge extraction algorithm, the method of locating the laser delamination area in the current image frame based on the image processing algorithm for detection and positioning includes: converting the current image frame into a grayscale image; performing edge detection on the grayscale image using a set edge extraction operator to obtain an edge sub-image, and performing adaptive thresholding on the edge sub-image to obtain a binary image; smoothing the binary image using a preset morphological processing method and removing useless areas that do not meet the set delamination area to obtain at least one area to be processed; and determining the laser delamination area based on the obtained area to be processed; Determining the laser delamination area based on the obtained area to be processed includes: extracting at least one maximum connected domain from the obtained area to be processed, and determining the wide center line of the circumscribed rectangle of the extracted maximum connected domain as the boundary of the laser delamination area; If the deep learning model is a trained semantic segmentation model, locating the laser delamination area in the current image frame based on the trained deep learning model for detection and positioning includes: inputting the current image frame into the trained semantic segmentation model, so that the semantic segmentation model identifies and outputs the laser delamination area block in the current image frame based on semantic features of the current image frame; If the deep learning model is a trained target detection model, locating the laser delamination area in the current image frame based on the trained deep learning model for detection and positioning includes: inputting the current image frame into the trained target detection model, so that the target detection model recognizes and outputs position information of the laser delamination area in the current image frame based on depth features of the current image frame; Determining a corresponding target correction scheme based on the current designated image characteristic parameters of the laser delamination area includes: searching for a characteristic parameter interval range in which the designated image characteristic parameters are located from preset characteristic parameter interval ranges; and determining a configured correction scheme for the characteristic parameter interval range as the target correction scheme; If the target correction scheme includes at least a brightness correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximate conditions includes: correcting the brightness of the laser delamination area using the brightness correction scheme so that the brightness of the corrected laser delamination area and the brightness of other image areas in the current image frame meet preset brightness approximate conditions; and / or, If the target correction scheme includes at least a color correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximation conditions includes: correcting the color of the laser delamination area using the color correction scheme so that the overall hue of the corrected laser delamination area and the hue of other image areas in the current image frame meet preset color approximation conditions; and / or, If the target correction scheme includes at least a boundary correction scheme, correcting the laser delamination area using the target correction scheme so that image parameters of the corrected laser delamination area and image parameters of other image areas in the current image frame meet preset approximation conditions includes: smoothing the boundary of the laser delamination area using the boundary correction scheme so that the corrected laser delamination area and other image areas in the current image frame meet preset fusion conditions; and / or If the target correction scheme includes at least an adaptive correction scheme, correcting the laser delamination area using the target correction scheme includes: correcting the laser delamination area using a trained deep learning network model for adaptive correction; the deep learning network model for adaptive correction is obtained by learning depth features of images around the laser delamination area; and / or, If the target correction scheme includes at least a previous frame reference correction scheme, the use of the target correction scheme to correct the laser stratification area so that the image parameters of the corrected laser stratification area and the image parameters of other image areas in the current image frame meet preset approximation conditions includes: obtaining a reference area at the same position as the laser stratification area from the previous frame image of the current image frame, and effectively fusing the reference area with the laser stratification area in the current image frame.

10. An image processing system for eliminating laser delamination, characterized in that: The system comprises: the device according to any one of claims 8 to 9, and a display terminal; The display end is used to display the current image frame; the original laser stratification area in the current image frame meets the preset approximation condition with other image areas in the current image frame after correction, and the preset approximation condition is used to make the appearance of the original laser stratification area in the current image frame after correction meet the approximate condition with the appearance of other image areas in the current image frame to eliminate the laser stratification in the current image frame.

11. An electronic device, characterized in that: The electronic device includes: a processor and a machine-readable storage medium; The machine-readable storage medium stores machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps of any one of claims 1-7.

Images