Image processing method, device and equipment

By determining and restoring the layered area of ​​the image in holmium laser surgery based on historical frames in endoscopic image processing, the visual discomfort caused by holmium laser layering phenomenon is solved, and the surgical visual perception and image display effect are improved.

CN119090777APending Publication Date: 2024-12-06HANGZHOU HAIKANG HUIYING TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411112598.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

During the holmium laser surgery, due to the large energy generated by the holmium laser, the holmium laser stratification phenomenon occurs in the images collected by the endoscopy, causing visual discomfort and interference in the field of operation, affecting the surgical effect.

Method used

By determining the hierarchical area of ​​the current frame based on the historical frame and restoring information, the area where information is lost is restored, the image display effect is improved and the doctor's visual impairment is weakened.

Benefits of technology

Effectively weaken the doctor's visual discomfort, provide a good surgical visual sense, improve image display effect, and ensure surgical effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090777A_ABST
    Figure CN119090777A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method, device and equipment, and the method comprises the steps: determining a hierarchical region of a current frame based on a historical frame before the current frame, the hierarchical region being an information loss region in the current frame, and the information loss region needing information recovery; performing information recovery on the hierarchical region of the current frame based on the historical frame; or performing information recovery on the hierarchical region of the current frame based on the current frame. According to the technical scheme, information recovery can be carried out on the information loss area of the current frame, the image display effect of the current frame is improved, and the visual perception of the current frame is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device and equipment. Background Art

[0002] Endoscopes are a commonly used medical device consisting of a light guide structure and a group of lenses. After the endoscope enters the interior of a target object, the endoscope can be used to capture images of the interior of the target object, and the target object can be inspected and treated based on the images of the interior of the target object.

[0003] Holmium laser is a solid-state pulsed laser. The energy generated by the holmium laser is absorbed by the body's tissues and has the effects of cutting, vaporization, and hemostasis. It has good penetration into stones of various properties without contacting or moving the stones, and causes relatively little damage to surrounding normal tissues. It can be used in conjunction with various endoscopes or puncture needles.

[0004] During holmium laser surgery, due to the high energy generated by the holmium laser, the flame generated by the interaction between the holmium laser and the body tissue will affect the normal exposure of the image, resulting in high brightness or even overexposure in some areas, and holmium laser stratification in the image. In summary, when an endoscope is used in combination with a holmium laser, when an image of the interior of the target object is collected through the endoscope, there will be holmium laser stratification areas in the image caused by the holmium laser.

[0005] In this case, the video will occasionally show dynamic "light bars", causing visual discomfort, affecting the operating field of view, interfering with the doctor's judgment, and resulting in failure to achieve the expected surgical effect. Summary of the invention

[0006] The present application provides an image processing method, the method comprising:

[0007] Determine a hierarchical region of the current frame based on a historical frame preceding the current frame, wherein the hierarchical region is an information loss region in the current frame, and the information loss region needs to be restored;

[0008] The information of the layered region of the current frame is restored based on the historical frame; or the information of the layered region of the current frame is restored based on the current frame.

[0009] The present application provides an image processing device, the device comprising:

[0010] A determination module, configured to determine a hierarchical region of the current frame based on a historical frame preceding the current frame, wherein the hierarchical region is an information loss region in the current frame and information recovery is required for the information loss region;

[0011] A processing module is used to restore information of the layered area of ​​the current frame based on the historical frame; or to restore information of the layered area of ​​the current frame based on the current frame.

[0012] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method of the above example of the present application.

[0013] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the image processing method of the above example of the present application is implemented.

[0014] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement the image processing method of the above example of the present application.

[0015] It can be seen from the above technical scheme that in the embodiment of the present application, the layered area of ​​the current frame can be determined based on the historical frame, and the information of the layered area can be restored based on the historical frame or the current frame to obtain the current frame after the information is restored. In this way, the information of the information lost area of ​​the current frame can be restored, so that no information loss occurs in the current frame, thereby improving the image display effect of the current frame and improving the visual experience of the current frame.

[0016] For example, if the current frame is an image of the inside of the target object acquired by an endoscope, and there is a stratified area caused by a strong light pulse in the current frame, the information of the stratified area caused by the strong light pulse is restored. By restoring the stratified area, the appearance of the stratified area is made close to that of a normal image, thereby reducing the physician's visual impairment, reducing the physician's visual discomfort, providing a good surgical perception, and achieving the expected surgical effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of an image processing method in one embodiment of the present application;

[0018] Figure 2A is a schematic diagram of the holmium laser delamination phenomenon in one embodiment of the present application;

[0019] Figure 2B It is a schematic diagram of the principle of the occurrence frequency of holmium laser stratification in one embodiment of the present application;

[0020] Figure 3 is a flowchart of an image processing method in one embodiment of the present application;

[0021] Figure 4A is a schematic diagram of determining the hierarchical region of the current frame using a template matching method;

[0022] Figure 4B is a schematic diagram of sliding windows for the current frame and the historical frame in one implementation of the present application;

[0023] Figure 4C is a schematic diagram of determining the hierarchical region of the current frame using a deep learning-based method;

[0024] Figure 4D It is a schematic diagram of the first generator network obtained by training with a generative adversarial network;

[0025] Figure 4E is a schematic diagram of determining the hierarchical region of the current frame using a deep learning-based method;

[0026] Figure 4F is a schematic diagram of determining the hierarchical region of the current frame using a deep learning-based method;

[0027] Figure 4G It is a schematic diagram of the second generator network obtained by training with a generative adversarial network;

[0028] Figure 5A It is a schematic diagram of using a multi-frame synthesis method to restore information of a layered area of ​​a current frame;

[0029] Figure 5B This is a schematic diagram of matching block search between a historical frame and a current frame in one implementation manner of the present application;

[0030] Figure 5C It is a schematic diagram of using a deep learning-based method to recover information from a layered region;

[0031] Figure 6 This is a schematic diagram of the real effect of the before and after images of holmium laser layering treatment;

[0032] Figure 7 is a schematic structural diagram of an image processing device in one embodiment of the present application;

[0033] Figure 8 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0034] In the embodiment of the present application, an image processing method is proposed, which can be applied to electronic devices. Figure 1 FIG. 1 is a flow chart of the image processing method, which may include:

[0035] Step 101: determine a hierarchical region of a current frame based on a historical frame preceding the current frame. The hierarchical region is an information loss region in the current frame, and information recovery is required for the information loss region.

[0036] Step 102: restore information of the layered region of the current frame based on the historical frame; or restore information of the layered region of the current frame based on the current frame.

[0037] In a possible implementation, the layered region of the current frame is determined based on the historical frames preceding the current frame, which may include but is not limited to: sliding the current frame to obtain multiple target windows; for each target window, obtaining a reference window corresponding to the target window from the historical frame, and determining the similarity between the reference window and the target window; based on the similarity, determining the target window as a layered window or a non-layered window. The current frame is divided into multiple regions to be analyzed, and each region to be analyzed includes K consecutive pixel rows, where K can be a positive integer; for each region to be analyzed, the number of layered windows in the region to be analyzed is counted; if the number is greater than a number threshold, the region to be analyzed is determined as a layered region.

[0038] For example, when the historical frame is used as the current frame, it can be determined whether there is a hierarchical region in the historical frame. If there is a hierarchical region in the historical frame, the hierarchical region of the historical frame is determined. In summary, the hierarchical region of the historical frame can be known, and then it can be known whether each reference window of the historical frame belongs to a hierarchical region or a non-hierarchical region.

[0039] Based on this, if the reference window corresponding to the target window belongs to the layered area, then if the similarity is greater than the first threshold, the target window is determined as a layered window, and if the similarity is not greater than the first threshold, the target window is determined as a non-layered window. Alternatively, if the reference window corresponding to the target window belongs to the non-layered area, then if the similarity is greater than the second threshold, the target window is determined as a non-layered window, and if the similarity is not greater than the second threshold, the target window is determined as a layered window.

[0040] In a possible implementation, determining the hierarchical region of the current frame based on the historical frame preceding the current frame may include, but is not limited to: inputting the historical frame to a trained first generator network, and outputting a simulated hierarchical image through the first generator network, wherein the simulated hierarchical image may include a simulated hierarchical region and a simulated normal region. The current frame is divided into a plurality of regions to be analyzed, and for each region to be analyzed, a reference region corresponding to the region to be analyzed is obtained from the simulated hierarchical image, wherein the position of the region to be analyzed in the current frame is the same as the position of the reference region in the simulated hierarchical image (e.g., the central pixel positions of the two regions are the same, and the widths of the two regions are the same, and the heights of the two regions are the same). If the reference region is a simulated hierarchical region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated hierarchical region is greater than a first similarity threshold, the region to be analyzed is determined as the hierarchical region of the current frame. If the reference region is a simulated normal region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated normal region is less than a second similarity threshold, the region to be analyzed is determined as the hierarchical region of the current frame.

[0041] In a possible implementation, determining the hierarchical region of the current frame based on the historical frame preceding the current frame may include, but is not limited to: inputting the historical frame to a trained second generator network, and outputting a simulated normal image through the second generator network, the simulated normal image may include a simulated normal region. The current frame is divided into a plurality of regions to be analyzed, and for each region to be analyzed, a reference region corresponding to the region to be analyzed is obtained from the simulated normal image, the position of the region to be analyzed in the current frame is the same as the position of the reference region in the simulated normal image; if the reference region is a simulated normal region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated normal region is less than a similarity threshold, the region to be analyzed is determined as the hierarchical region of the current frame.

[0042] In a possible implementation, information recovery of the hierarchical region of the current frame based on the historical frame may include but is not limited to: dividing the non-hierarchical region of the current frame into multiple image blocks; for each image block, determining a matching block of the image block from the historical frame, wherein the matching block is the image block in the historical frame that is most similar to the image block. Determine a target motion vector based on the relative displacement between each image block and the matching block of the image block; for each target pixel point in the hierarchical region of the current frame, determine a reference pixel point corresponding to the target pixel point from the historical frame based on the target motion vector; and restore information of the target pixel point based on the pixel value of the reference pixel point. For example, the pixel value of the reference pixel point may be used as the pixel value of the target pixel point, thereby restoring information of the target pixel point.

[0043] In a possible implementation, performing information recovery on the hierarchical region of the current frame based on the current frame may include, but is not limited to: inputting the current frame and the hierarchical region position information into a trained information recovery network, wherein the hierarchical region position information may represent the position of the hierarchical region in the current frame; simulating and generating correction and repair information through the information recovery network; determining the hierarchical region based on the hierarchical region position information through the information recovery network; and performing information recovery on the hierarchical region through the correction and repair information.

[0044] The correction and restoration information includes the pixel value of each pixel point in the layered area, so that the pixel value of each pixel point can be assigned to the layered area, thereby restoring the information of the layered area.

[0045] Exemplarily, an image of the target object can be acquired through an endoscope to obtain a historical frame and a current frame, wherein when the current frame is acquired through the endoscope, a strong light pulse appears in the environment where the target object is located, and the strong light pulse can be a strong light pulse generated by a holmium laser or a strong light pulse generated by other means. Based on this, the stratified area can include a holmium laser stratified area caused by a holmium laser pulse.

[0046] In addition, the historical frame may be an adjacent frame preceding the current frame; or, if an adjacent frame preceding the current frame has a stratified region, the historical frame may be an adjacent frame after the stratified region is restored by information, or, the historical frame may be an adjacent frame before the stratified region is restored by information; or, the historical frame may be the first frame preceding the current frame in which no stratified region exists. When the current frame is acquired through an endoscope, a strong light pulse causes a stratified region (such as a holmium laser stratified region) to exist in the current frame, and the stratified region may be a strip-shaped region, the stratified region has a lateral edge, and the brightness of the stratified region is higher than the brightness of the non-stratified region.

[0047] It can be seen from the above technical scheme that in the embodiment of the present application, the layered area of ​​the current frame can be determined based on the historical frame, and the information of the layered area can be restored based on the historical frame or the current frame to obtain the current frame after the information is restored. In this way, the information of the information lost area of ​​the current frame can be restored, so that no information loss occurs in the current frame, thereby improving the image display effect of the current frame and improving the visual experience of the current frame.

[0048] For example, if the current frame is an image of the inside of the target object captured by an endoscope, and there is a stratified area caused by a strong light pulse in the current frame, the information of the stratified area produced by the strong light pulse is restored. By restoring the stratified area, the appearance of the stratified area is made close to that of a normal image, thereby reducing the physician's visual impairment, reducing the physician's visual discomfort, providing a good surgical perception, and achieving the expected surgical effect.

[0049] The technical solution of the embodiment of the present application is described below in conjunction with specific application scenarios.

[0050] Before introducing the technical solutions of the embodiments of the present application, technical terms related to the present application are first introduced.

[0051] Endoscopes: Endoscopes are a commonly used medical device, consisting of a light guide structure and a set of lenses. After the endoscope enters the target object (such as a patient waiting for a specimen), the endoscope can be used to collect images of a specified type of tissue inside the target object, and the target object can be inspected and treated based on the image. The specified type of tissue can be any tissue inside the target object, that is, the target object's in vivo tissue, such as intracavitary tissue, intraauricular tissue, intranasal tissue, intralarynx tissue, nerve tissue, organ tissue, etc. Of course, the above are just a few examples, and there is no restriction on the type of the specified type of tissue.

[0052] Holmium laser: Holmium laser is a solid-state pulsed laser with a wavelength of 2100nm, which is an invisible light. The energy generated by holmium laser is absorbed by the body tissue and has the effects of cutting, vaporization, and hemostasis. It can penetrate stones of various properties well without contacting and moving stones, and causes relatively little damage to surrounding normal tissues. Holmium laser can be transmitted by optical fiber and can be used in conjunction with various endoscopes or puncture needles.

[0053] Holmium laser stratification: When holmium laser pulses are emitted, the image will show continuous "fault" phenomena, which are defined as holmium laser stratification. For example, during holmium laser surgery, the interaction between the laser and the stone can generate visible flames. Within a frame of exposure time, the release time (pulse width) of the holmium laser is often very short (hundreds of microseconds), resulting in a "response" in some exposure lines, while the remaining exposure lines are still exposed normally, thus forming holmium laser stratification. Holmium laser stratification has relatively clear horizontal edges, and the brightness of holmium laser stratification is significantly higher than that of other areas of the image.

[0054] See also Figure 2A The figure shows the schematic diagram of the holmium laser delamination phenomenon. The left side is the normal exposure image, and the right side is the holmium laser delamination image. In the holmium laser delamination image, there are the normal exposure area, the holmium laser delamination area, and the horizontal edge of the holmium laser delamination. It can be seen from the holmium laser delamination image that the holmium laser delamination has a relatively clear horizontal edge, and the brightness of the holmium laser delamination is significantly higher than that of other areas of the image.

[0055] For holmium laser stratification, since the stratification width in a single frame is determined by the laser pulse width and exposure "slope" of the holmium laser, and the number of stratification phenomena in consecutive frames is closely related to the excitation frequency of the holmium laser, the area and position of the holmium laser stratification are not fixed in consecutive frames.

[0056] See also Figure 2B As shown, this is a schematic diagram of the principle of the frequency of occurrence of holmium laser stratification.

[0057] For the image generation time, when the image generation time is a high level, it means that the image will be captured at the high level time, and when the image generation time is a low level, it means that the image will not be captured at the low level. For the holmium laser excitation frequency at frequency 1, when the holmium laser excitation frequency is a high level, it means that a holmium laser pulse will be emitted, resulting in the image area corresponding to the high level time being the holmium laser stratification area, and when the holmium laser excitation frequency is a low level, it means that a holmium laser pulse will not be emitted, resulting in the image area corresponding to the low level time being the normal exposure area. Similarly, for the holmium laser excitation frequency at frequency 2, when the holmium laser excitation frequency is a high level, it means that a holmium laser pulse will be emitted, resulting in the image area corresponding to the high level time being the holmium laser stratification area, and when the holmium laser excitation frequency is a low level, it means that a holmium laser pulse will not be emitted, resulting in the image area corresponding to the low level time being the normal exposure area.

[0058] For example, holmium laser is a highly coherent radiation source, and is used as a surgical tool for its advantages such as less thermal damage and less bleeding. The working principle of holmium laser is: using water as an absorption medium, the energy generated vaporizes the water between tissues, forming tiny cavitation bubbles, which transmit energy to the tissues and crush the diseased tissues into powder. The energy generated by holmium laser is just in the strongest energy absorption range of water. When holmium laser acts on the diseased tissue, the surrounding water and body fluids absorb a large amount of excess energy, reducing damage to the surrounding tissues.

[0059] During holmium laser surgery, due to the high energy generated by the holmium laser, the flame generated by the interaction between the holmium laser and the body tissue will affect the normal exposure of the image, resulting in high brightness or even overexposure in some areas, and holmium laser stratification in the image. The size and position of holmium laser stratification are not fixed, and are related to the holmium laser frequency, the structure of the lesion tissue, and the distance from the lesion tissue.

[0060] In summary, when an endoscope is used in combination with a holmium laser, when an image of the interior of a target object is collected through an endoscope, there will be a holmium laser stratification area in the image caused by the holmium laser. In this case, the surgical process video viewed by the physician during the operation will occasionally show dynamic "light bars", causing visual discomfort, affecting the operating field, interfering with the physician's judgment, and resulting in failure to achieve the expected surgical effect.

[0061] In view of the above findings, an image processing method is proposed in an embodiment of the present application, which can repair holmium laser stratification to make its appearance close to that of a normal image, reduce the visual impairment of the physician, and provide a good surgical viewing experience.

[0062] See also Figure 3 As shown, it is a flowchart of the image processing method. After obtaining the video stream, each image frame in the video stream is traversed in turn. For the image frame currently traversed, the image frame can be called the current frame. On this basis, the layered area of ​​the current frame can be detected and located. If the current frame has a layered area, the layered area can be corrected and repaired, and an image with the layered area eliminated can be output.

[0063] Exemplarily, for the detection and positioning process of the layered area, the layered area of ​​the current frame can be determined based on the historical frame before the current frame (i.e., the image frame located before the current frame in the video stream). The layered area may be an information loss area within the current frame, and the information loss area needs to be restored.

[0064] For example, the layered area is an area that needs to display normal pixel values ​​(such as pixel values ​​that are not overexposed, etc.), but due to some reasons, the layered area does not display normal pixel values, that is, the information of the layered area is lost and the normal pixel values ​​cannot be displayed. Moreover, it is necessary to restore the information of the pixel values ​​of the layered area so that the layered area displays normal pixel values, that is, to obtain normal pixel values ​​by restoring the information of the layered area.

[0065] The region satisfying the above conditions may be called a hierarchical region, that is, the hierarchical region is an information loss region in the current frame (ie, information loss in the hierarchical region), and the information in the hierarchical region needs to be restored.

[0066] For example, the current frame is an image of the interior of the target object acquired through an endoscope. When acquiring the current frame through the endoscope, if a holmium laser pulse is emitted inside the target object, the stratified area may include a holmium laser stratified area caused by the holmium laser pulse. Of course, in addition to acquiring the image of the interior of the target object through an endoscope, other methods may be used to acquire the current frame, and when acquiring the current frame, a holmium laser pulse is emitted inside the target object, causing the current frame to include a holmium laser stratified area caused by the holmium laser pulse.

[0067] Of course, in addition to the holmium laser stratification area caused by the holmium laser pulse, other types of stratification areas may also exist. In this embodiment, there is no restriction on this stratification area, as long as the stratification area exists in the current frame, and the stratification area is the information loss area in the current frame, and the stratification area needs to recover information. For the convenience of description, the holmium laser stratification area caused by the holmium laser pulse is taken as an example. Based on this, the video stream includes multiple frames of images, and each frame of the image is an image of the inside of the target object collected by an endoscope. In addition, when collecting images through an endoscope, holmium laser pulses may be emitted inside the target object, resulting in the presence of holmium laser stratification areas in some image frames, and no holmium laser stratification areas in the remaining image frames.

[0068] In summary, the characteristic of the stratified area is that when the current frame is collected through the endoscope, a strong light pulse appears in the environment where the target object is located. The strong light pulse can be a strong light pulse generated by a holmium laser or a strong light pulse generated by other means. In this way, when the current frame is collected through the endoscope, the strong light pulse will cause the current frame to have a stratified area (such as a holmium laser stratified area). The stratified area can be a strip-shaped area with a lateral edge, and the brightness of the stratified area is higher than that of the non-stratified area.

[0069] In this application scenario, for the detection and positioning process of the layered area, it is possible to determine whether the current frame has a holmium laser layered area based on the historical frames before the current frame. If there is no holmium laser layered area, it means that the current frame is an image frame without a holmium laser layered area, and there is no need to correct and repair the current frame.

[0070] If there is a holmium laser stratification area, it means that the current frame is an image frame with a holmium laser stratification area, and the current frame needs to be corrected and repaired. In order to correct and repair the current frame, it is also necessary to determine the holmium laser stratification area of ​​the current frame, that is, locate the position of the holmium laser stratification area from the current frame, such as when the holmium laser stratification area is a rectangular area, determine the four angular coordinates of the holmium laser stratification area, or determine one angular coordinate of the holmium laser stratification area, as well as the width and height of the holmium laser stratification area.

[0071] Exemplarily, in the correction and repair process of the layered region, the information of the layered region of the current frame can be restored based on the historical frame (i.e., the image frame located before the current frame in the video stream); or, the information of the layered region of the current frame can be restored based on the current frame. For example, taking the holmium laser layered region as an example, after locating the position of the holmium laser layered region in the current frame, the information of the holmium laser layered region in the current frame can be restored based on the historical frame, or the information of the holmium laser layered region in the current frame can be restored based on the current frame, so as to obtain the current frame after the information is restored.

[0072] In the above process, the holmium laser layered area is taken as an example to introduce the detection and positioning process of the layered area and the correction and repair process of the layered area. However, in actual scenarios, it is not limited to the holmium laser layered area. As long as there is a layered area in the current frame, and the layered area is an information loss area, and the layered area needs to recover information, the image processing method of this embodiment can be used to detect and position the layered area in the current frame, and when there is a layered area in the current frame, the layered area is corrected and repaired. The following describes the detection and positioning process of the layered area and the correction and repair process of the layered area in combination with specific application scenarios.

[0073] First, the detection and positioning process of the layered area.

[0074] Exemplarily, after obtaining the video stream, it is necessary to traverse each image frame in the video stream in turn. For the currently traversed image frame, the image frame is called the current frame, and the layered area detection and positioning can be performed on the current frame. The detection of the layered area refers to detecting whether there is a layered area in the current frame. If there is no layered area, the correction and repair process is not performed on the current frame. The positioning of the layered area means that if there is a layered area in the current frame, the position of the layered area is located from the current frame, and then the correction and repair process is performed on the current frame based on the position of the layered area. For the detection and positioning process of the layered area, according to the characteristics of the layered area (according to the holmium laser layered area), the layered area of ​​the current frame can be determined based on the historical frames before the current frame. For example, the layered area of ​​the current frame can be determined by using a template matching method.

[0075] For example, see Figure 4A As shown, it is a schematic diagram of determining the hierarchical area of ​​the current frame by using the template matching method. The process may include: image frame acquisition, similarity calculation and hierarchical area detection and positioning.

[0076] For the image frame acquisition process, the video stream can be acquired, and the current frame and the historical frames before the current frame can be acquired from the video stream. For example, each image frame in the video stream needs to be traversed in sequence, the currently traversed image frame is used as the current frame, and the historical frames before the current frame are acquired from the video stream.

[0077] For example, the historical frame may be an adjacent frame preceding the current frame. Assuming that the current frame is the 10th frame in the video stream, the historical frame preceding the current frame is the 9th frame in the video stream, that is, a frame preceding the current frame.

[0078] Alternatively, if the adjacent frame before the current frame has a layered region, the historical frame can be the adjacent frame after the layered region is restored, or the historical frame can be the adjacent frame before the layered region is restored. Assume that the current frame is the 10th frame in the video stream, and the 9th frame in the video stream has a layered region (when the 9th frame is used as the current frame, it can be determined whether the 9th frame has a layered region, and the information of the layered region of the 9th frame can be restored). Based on this, the 9th frame after the layered region is restored can be used as the historical frame before the current frame, or the 9th frame before the layered region is restored can be used as the historical frame before the current frame.

[0079] Alternatively, the historical frame may be the first frame before the current frame that does not have a layered region. Assuming that the current frame is the 10th frame in the video stream, first determine whether the 9th frame in the video stream has a layered region. If not, the historical frame before the current frame is the 9th frame in the video stream. If so, continue to determine whether the 8th frame in the video stream has a layered region. If not, the historical frame before the current frame is the 8th frame in the video stream. If so, continue to determine whether the 7th frame in the video stream has a layered region, and so on.

[0080] In summary, the historical frame before the current frame can be obtained, and the historical frame is taken as an example of a frame image. Of course, the historical frame can also be multiple frames of images, and the processing method of multiple frames of images will not be repeated.

[0081] For the similarity calculation process, the current frame can be slid to obtain multiple target windows. For example, an M*N window can be used to slide the current frame, and the current frame can be slid from left to right and from top to bottom. In addition, the historical frame can be slid to obtain multiple historical windows. For example, an M*N window can be used to slide the historical frame, and the historical frame can be slid from left to right and from top to bottom. See Figure 4B , which is a schematic diagram of sliding windows for the current frame and the historical frames.

[0082] For each target window in the current frame, a reference window corresponding to the target window can be obtained from the historical frame (the historical window corresponding to the target window can be called a reference window). Obviously, the position of the target window in the current frame can be the same as the position of the reference window in the historical frame.

[0083] Then, the similarity (similarity) between the reference window and the target window can be determined. When the similarity is greater, it means that the reference window is more similar to the target window, and when the similarity is smaller, it means that the reference window is less similar to the target window. When determining the similarity between the reference window and the target window, it can be achieved by normalized cross-correlation, square difference, etc., and there is no limitation on this determination method.

[0084] Then, based on the similarity, the target window is determined as a layered window or a non-layered window. For example, the scene of the holmium laser is fixed during use, and the historical frame and the current frame are both images of a specified type of organization inside the target object. The similarity between the historical frame and the current frame should be very high. Based on this, based on the similarity, the target window can be determined as a layered window or a non-layered window in the following manner.

[0085] If the reference window does not belong to the layered area, then when the target window belongs to the layered area, the information of the target window and the reference window has obvious differences, that is, the similarity between the target window and the reference window is relatively small. When the target window does not belong to the layered area, the information of the target window and the reference window has no obvious differences, that is, the similarity between the target window and the reference window is relatively large.

[0086] Based on this, if the similarity is less than the similarity threshold, the target window is determined as a layered window. On the contrary, if the similarity is not less than the similarity threshold, the target window is determined as a non-layered window.

[0087] If the reference window belongs to a layered area, then when the target window does not belong to a layered area, there is an obvious difference in information between the target window and the reference window, that is, the similarity between the target window and the reference window is relatively large; when the target window belongs to a layered area, there is no obvious difference in information between the target window and the reference window, that is, the similarity between the target window and the reference window is relatively small.

[0088] Based on this, if the similarity is greater than the similarity threshold, the target window is determined as a non-layered window. On the contrary, if the similarity is not greater than the similarity threshold, the target window is determined as a layered window.

[0089] From the above, it can be seen that in the similarity calculation process, for each target window in the current frame, the target window can be determined as a layered window, or the target window can be determined as a non-layered window, that is, all layered windows and all non-layered windows can be found from the current frame.

[0090] For the hierarchical region detection and positioning process, the current frame can be divided into multiple regions to be analyzed, each of which includes K consecutive pixel rows, and K is a positive integer. K can be configured based on experience, such as 4, 5, 6, etc., and there is no restriction on this. For example, taking 6 pixel rows as an example, the 1st to 6th pixel rows constitute the region a1 to be analyzed, the 7th to 12th pixel rows constitute the region a2 to be analyzed, and so on.

[0091] For each region to be analyzed, the number of layered windows in the region to be analyzed can be counted. For example, if all layered windows are located in the region to be analyzed, or if part of the layered windows are located in the region to be analyzed, the number of layered windows in the region to be analyzed is increased by 1. On this basis, if the number of layered windows in the region to be analyzed is greater than a quantity threshold, the region to be analyzed is determined as a layered region. If the number is not greater than the quantity threshold, the region to be analyzed is determined as a non-layered region.

[0092] For example, based on the working principle of holmium laser, when there is a stratified area caused by holmium laser in the current frame, the entire pixel row is the stratified area caused by holmium laser, and not part of the pixel area in the pixel row is the stratified area. Based on this, if the number of stratified windows in the area to be analyzed is not greater than the quantity threshold, it means that only part of the area to be analyzed is a stratified window, and these stratified windows may be the result of false detection, and the area to be analyzed should not be determined as a stratified area. If the number of stratified windows in the area to be analyzed is greater than the quantity threshold, it means that most of the area to be analyzed is a stratified window, and the area to be analyzed should be determined as a stratified area. In summary, if the number of stratified windows in the area to be analyzed is greater than the quantity threshold, the area to be analyzed is determined as a stratified area. Otherwise, the area to be analyzed is determined as a non-stratified area.

[0093] At this point, the detection and positioning process of the layered region is completed, and it can be determined whether there is a layered region in the current frame. If so, the layered region of the current frame can be determined based on the historical frames before the current frame.

[0094] Second, the detection and positioning process of the layered area.

[0095] Exemplarily, after obtaining the video stream, each image frame in the video stream is traversed in sequence. For the currently traversed image frame, the image frame is called the current frame, and the layered area detection and positioning is performed on the current frame. The detection of the layered area refers to detecting whether there is a layered area in the current frame, and the positioning of the layered area refers to locating the position of the layered area from the current frame if there is a layered area in the current frame. For the detection and positioning process of the layered area, the layered area of ​​the current frame can be determined based on the historical frames before the current frame. For example, based on the historical frames, a deep learning-based method can be used to determine the layered area of ​​the current frame.

[0096] For example, see Figure 4C As shown, it is a schematic diagram of using a deep learning-based method to determine the hierarchical area of ​​the current frame. The process may include: image frame acquisition process, generator processing process, contrast detection process and hierarchical area detection and positioning process. The processing of these processes is explained below.

[0097] For the image frame acquisition process, the video stream can be acquired, and the current frame and the historical frames before the current frame can be acquired from the video stream. For example, each image frame in the video stream needs to be traversed in sequence, the currently traversed image frame is used as the current frame, and the historical frames before the current frame are acquired from the video stream.

[0098] For example, the historical frame may be an adjacent frame preceding the current frame. Alternatively, if the adjacent frame preceding the current frame has a layered region, the historical frame may be an adjacent frame after the layered region is restored, or the historical frame may be an adjacent frame before the layered region is restored. Alternatively, the historical frame may be the first frame preceding the current frame that does not have a layered region. The above-mentioned historical frame scheme may refer to the above-mentioned embodiment.

[0099] For the generator processing process, the historical frame (one or more historical frames) can be input into the trained first generator network (see the subsequent embodiments for the training process of the first generator network).

[0100] For the first generator network, the first generator network can be a deep learning network, and the first generator network is used to generate a simulated layered image (i.e., simulate and generate a frame of image based on input data), and the simulated layered image needs to include a simulated layered area and a simulated normal area, that is, the simulated layered image is an image with a simulated layered area. Regarding the process of the first generator network generating the simulated layered image, there is no limitation in this embodiment, as long as the first generator network can generate the simulated layered image.

[0101] Based on this, after the historical frame is input into the first generator network, the first generator network can generate a frame of simulated layered image based on the historical frame simulation, and the first generator network can output the simulated layered image, and the simulated layered image can include a simulated layered area and a simulated normal area.

[0102] The simulated layered image is an image predicted and generated by the first generator network, and represents a simulated layered image of the current frame predicted based on historical frames, that is, a layered image that may appear in the current frame.

[0103] For the contrast detection process, the current frame can be divided into multiple areas to be analyzed, each of which includes K consecutive pixel rows, where K is a positive integer and can be configured based on experience, such as 4, 5, 6, etc.

[0104] For each area to be analyzed in the current frame, a reference area corresponding to the area to be analyzed can be obtained from the simulated layered image, and the reference area is also K pixel rows in the simulated layered image. Obviously, the position of the area to be analyzed in the current frame and the position of the reference area in the simulated layered image can be the same (such as the central pixel position of the two areas is the same, and the width and height of the two areas are the same).

[0105] The similarity (similarity) between the region to be analyzed and the reference region can be determined. The greater the similarity, the more similar the region to be analyzed is to the reference region. The smaller the similarity, the less similar the region to be analyzed is to the reference region. When determining the similarity between the region to be analyzed and the reference region, it can be achieved by normalized cross-correlation, square difference, etc., and there is no limitation on this determination method.

[0106] For the layered area detection and positioning process, for each area to be analyzed in the current frame, if the reference area corresponding to the area to be analyzed is a simulated layered area in the simulated layered image, that is, the area to be analyzed corresponds to the simulated layered area, then: if the similarity between the area to be analyzed and the simulated layered area is greater than the first similarity threshold, that is, the area to be analyzed is similar to the simulated layered area, since the simulated layered area is the layered area of ​​the simulated layered image, the area to be analyzed is determined as the layered area of ​​the current frame.

[0107] If the similarity between the area to be analyzed and the simulated layered area is not greater than the first similarity threshold, that is, the area to be analyzed and the simulated layered area are not similar, since the simulated layered area is the layered area of ​​the simulated layered image, the area to be analyzed can be determined as a non-layered area of ​​the current frame.

[0108] For each area to be analyzed in the current frame, if the reference area corresponding to the area to be analyzed is a simulated normal area in the simulated layered image, that is, the area to be analyzed corresponds to the simulated normal area, then: if the similarity between the area to be analyzed and the simulated normal area is less than the second similarity threshold, that is, the area to be analyzed is not similar to the simulated normal area, since the simulated normal area is a normal area of ​​the simulated layered image (that is, a non-layered area), therefore, the area to be analyzed is determined as the layered area of ​​the current frame.

[0109] If the similarity between the area to be analyzed and the simulated normal area is not less than the second similarity threshold, that is, the area to be analyzed is similar to the simulated normal area, since the simulated normal area is the normal area of ​​the simulated layered image, the area to be analyzed can be determined as the non-layered area of ​​the current frame.

[0110] At this point, the detection and positioning process of the layered area is completed, and the layered area of ​​the current frame can be determined.

[0111] In a possible implementation, training data may be obtained, and the first generator network may be trained based on the training data. For example, based on images of different exposure conditions (layering with different temporal rules, etc.) generated by holmium lasers at different frequencies, these images include normal images without layered areas and layered images with layered areas, training data may be obtained, and the training data includes historical samples and sample labels, and a layered image may be used as a sample label, and a normal image before the layered image may be used as a historical sample.

[0112] See also Figure 4D As shown, it is a schematic diagram of obtaining the first generator network by training with a generative adversarial network.

[0113] The historical samples in the training data can be input to the generator (i.e., the model to be trained), and the generator is used to generate a simulated layered image, and the simulated layered image needs to include a simulated layered area and a simulated normal area, that is, the simulated layered image is an image with a simulated layered area. Based on this, after the historical samples are input to the generator, the generator can generate a frame of simulated layered image based on the historical samples, and output the simulated layered image, and the simulated layered image can include a simulated layered area and a simulated normal area.

[0114] Then, the sample labels in the training data and the simulated layered image can be input into the discriminator, and the discriminator can compare the similarity between the sample labels and the simulated layered image, and calculate the loss value based on the similarity between the sample labels and the simulated layered image. Obviously, the greater the similarity, the smaller the loss value, and vice versa, the smaller the similarity, the larger the loss value. Then, the network parameters of the generator can be adjusted based on the loss value, such as by using the gradient descent method to adjust the network parameters of the generator. The adjustment goal is to make the loss value smaller and smaller, and there is no restriction on this adjustment process.

[0115] If the adjusted generator has converged, the adjusted generator is output, that is, the adjusted generator is used as the first generator network. If the adjusted generator has not converged, the operation of inputting historical samples in the training data into the generator is returned, and so on, until the adjusted generator converges and the adjusted generator is output.

[0116] Third, the detection and positioning process of the layered area.

[0117] Exemplarily, after obtaining the video stream, each image frame in the video stream is traversed in sequence. For the currently traversed image frame, the image frame is called the current frame, and the layered region detection and positioning of the current frame is performed. For the detection and positioning process of the layered region, the layered region of the current frame can be determined based on the historical frames before the current frame, such as based on the historical frames, the layered region of the current frame can be determined by a deep learning-based method.

[0118] For example, see Figure 4E As shown, it is a schematic diagram of using a deep learning-based method to determine the hierarchical area of ​​the current frame. The process may include: image frame acquisition process, generator processing process, contrast detection process and hierarchical area detection and positioning process. The processing of these processes is explained below.

[0119] For the image frame acquisition process, a video stream is acquired, and the current frame and the historical frames before the current frame are acquired from the video stream. The historical frame is the adjacent frame before the current frame. Or, if the adjacent frame before the current frame has a layered region, the historical frame is the adjacent frame after the layered region is restored by information, or the historical frame is the adjacent frame before the layered region is restored by information. Or, the historical frame is the first frame before the current frame that does not have a layered region.

[0120] For the generator processing process, the historical frame (one or more historical frames) can be input into the trained second generator network (see the subsequent embodiments for the training process of the second generator network).

[0121] For the second generator network, the second generator network can be a deep learning network, and the second generator network is used to generate a simulated normal image (i.e., simulate and generate a frame of image based on input data), and the simulated normal image only includes a simulated normal area, that is, the simulated normal image is an image without a simulated layered area. Regarding the process of the second generator network generating the simulated normal image, this embodiment does not limit it.

[0122] Based on this, after the historical frame is input into the second generator network, the second generator network can generate a frame of simulated normal image based on the historical frame simulation, and the second generator network can output the simulated normal image, and the simulated normal image can include a simulated normal area, but does not include a simulated layered area.

[0123] The simulated normal image is an image predicted and generated by the second generator network, and represents a simulated normal image of the current frame predicted based on historical frames, that is, a normal image that may appear in the current frame.

[0124] For the contrast detection process, the current frame is divided into multiple areas to be analyzed, each of which includes K consecutive pixel rows. For each area to be analyzed in the current frame, a reference area corresponding to the area to be analyzed is obtained from the simulated normal image, and the reference area is also K pixel rows in the simulated normal image. The position of the area to be analyzed in the current frame is the same as the position of the reference area in the simulated normal image.

[0125] The similarity (similarity) between the region to be analyzed and the reference region can be determined. The greater the similarity, the more similar the region to be analyzed is to the reference region. The smaller the similarity, the less similar the region to be analyzed is to the reference region. When determining the similarity between the region to be analyzed and the reference region, it can be achieved by normalized cross-correlation, square difference, etc., and there is no limitation on this determination method.

[0126] For the layered region detection and positioning process, for each region to be analyzed in the current frame, if the reference region corresponding to the region to be analyzed is a simulated normal region in a simulated normal image, that is, the region to be analyzed corresponds to the simulated normal region, then: if the similarity between the region to be analyzed and the simulated normal region is less than the similarity threshold, that is, the region to be analyzed is not similar to the simulated normal region, the region to be analyzed is determined as the layered region of the current frame. If the similarity between the region to be analyzed and the simulated normal region is not less than the similarity threshold, that is, the region to be analyzed is similar to the simulated normal region, the region to be analyzed is determined as the non-layered region of the current frame. At this point, the layered region detection and positioning process is completed, and the layered region of the current frame is obtained.

[0127] In one possible implementation, see Figure 4F As shown in FIG. 1 , it is a schematic diagram of determining the hierarchical region of the current frame using a deep learning-based method. The process may include: image frame acquisition process, generator processing process, comparison detection process, and hierarchical region detection and positioning process. Figure 4E In contrast, in the image frame acquisition process, only the current frame needs to be acquired, and the historical frames before the current frame do not need to be acquired. In the generator processing process, the current frame is input to the second generator network instead of the historical frames. Figure 4F The rest of the process is Figure 4E Similar, no need to repeat it here.

[0128] In a possible implementation, training data may be obtained, and the second generator network may be trained based on the training data. For example, the training data includes historical samples and sample labels. A normal image (i.e., a non-layered image) may be used as a sample label, and the image preceding the normal image may be used as a historical sample. Figure 4G As shown, it is a schematic diagram of obtaining the second generator network by training with a generative adversarial network.

[0129] The historical samples in the training data can be input to the generator (i.e., the model to be trained), and the generator is used to generate a simulated normal image, and the simulated normal image only includes a simulated normal area. Based on this, after the historical samples are input to the generator, the generator can generate a frame of simulated normal image based on the historical samples, and output the simulated normal image, and the simulated normal image can include the simulated normal area.

[0130] The sample labels in the training data and the simulated normal image are input to the discriminator, which compares the similarity between the sample labels and the simulated normal image, and calculates the loss value based on the similarity between the sample labels and the simulated normal image. The greater the similarity, the smaller the loss value, and vice versa. The smaller the similarity, the larger the loss value. Then, the network parameters of the generator are adjusted based on the loss value.

[0131] If the adjusted generator has converged, the adjusted generator is output, that is, the adjusted generator is used as the second generator network. If the adjusted generator has not converged, the operation of inputting historical samples in the training data into the generator is returned, and so on, until the adjusted generator converges and the adjusted generator is output.

[0132] Fourth, the correction and repair (also called information recovery) process of the delaminated area.

[0133] Exemplarily, if there is a layered area in the current frame, the layered area (such as the holmium laser layered area) of the current frame can be corrected and repaired (information restored) to make the current frame close to a normal image (without layered area), thereby performing the holmium laser layered area elimination process on the current frame. For example, information restoration can be performed on the layered area of ​​the current frame based on the historical frame before the current frame. When restoring information on the layered area based on the historical frame, a multi-frame synthesis method can be used to restore information on the layered area, or other methods can be used to restore information on the layered area. The multi-frame synthesis method is used as an example below.

[0134] For example, see Figure 5A As shown, it is a schematic diagram of using a multi-frame synthesis method to restore information of the layered area of ​​the current frame. The process may include: an image frame acquisition process, a motion vector determination process, and a layered area correction and repair process. The processing of these processes is explained below.

[0135] For the image frame acquisition process, a video stream is acquired, and the current frame and the historical frames before the current frame are acquired from the video stream. If the adjacent frame before the current frame has a layered region, the historical frame is the adjacent frame after the layered region is restored by information. Or, the historical frame is the first frame before the current frame that does not have a layered region.

[0136] For the motion vector determination process, the non-hierarchical area of ​​the current frame (ie, other areas except the hierarchical area) may be divided into a plurality of non-overlapping image blocks, such as 4*4, 8*8 image blocks, and the like.

[0137] For each image block, determine the matching block of the image block from the historical frame. The matching block is the image block in the historical frame that is most similar to the image block. For example, the historical frame is divided into multiple image blocks (for the convenience of distinction, these image blocks are called reference blocks), and the size of the reference block is the same as the size of the image block of the current frame. The similarity between the image block of the current frame and each reference block in the historical frame is calculated in turn, and the reference block with the greatest similarity is used as the matching block of the image block. Obviously, the matching block of each image block can be obtained, and the size of the matching block is the same as the size of the image block, such as blocks of size A*B.

[0138] For each image block, the relative displacement between the image block and the matching block can be determined. The relative displacement indicates the positional deviation between the center of the image block and the center of the matching block. For example, the pixel coordinates of the center of the matching block in the historical frame are (3, 3), and the pixel coordinates of the center of the image block in the current frame are (4, 5). Then the relative displacement between the image block and the matching block is (1, 2), indicating that the image block is offset to the right by 1 pixel in the horizontal coordinate direction and offset to the upward by 2 pixels in the vertical coordinate direction. If the horizontal coordinate of the relative displacement is a positive value, it indicates an offset to the right; if the horizontal coordinate is a negative value, it indicates an offset to the left; if the vertical coordinate is a positive value, it indicates an offset to the upward; if the vertical coordinate is a negative value, it indicates a offset to the downward.

[0139] Obviously, the relative displacement of each image block (i.e., the relative displacement between the image block and the matching block of the image block) can be obtained, and based on the relative displacement of all image blocks, the target motion vector (also called motion reference vector) can be determined. For example, the average value of the relative displacements of all image blocks is used as the target motion vector. The number of occurrences of each relative displacement can be counted, and the relative displacement with the maximum number of occurrences is used as the target motion vector. Of course, the above is only an example, and there is no limitation on the method of determining the target motion vector.

[0140] See also Figure 5B As shown, it is a schematic diagram of searching for matching blocks between a historical frame and a current frame. A matching block of an image block can be determined from a historical frame. The matching block is an image block in the historical frame that is most similar to the image block.

[0141] In the correction and repair process of the layered area, for each target pixel point in the layered area of ​​the current frame, the reference pixel point corresponding to the target pixel point can be determined from the historical frame based on the target motion vector. For example, the first pixel point corresponding to the target pixel point is first determined from the historical frame, and the pixel coordinates of the target pixel point in the current frame are the same as the pixel coordinates of the first pixel point in the historical frame. Then, based on the first pixel point, the target motion vector can be used to offset the first pixel point to obtain the reference pixel point in the historical frame, that is, the reference pixel point corresponding to the target pixel point.

[0142] After obtaining the reference pixel corresponding to the target pixel, the information of the target pixel can be restored based on the pixel value of the reference pixel (i.e., the pixel value of the reference pixel in the historical frame), that is, the pixel value of the target pixel in the current frame can be restored. For example, the pixel value of the reference pixel can be used as the pixel value of the target pixel in the current frame, thereby restoring the information of the target pixel. Obviously, after performing the above processing on each target pixel in the layered area, the information of the layered area can be restored, thereby achieving correction and repair of the layered area.

[0143] At this point, the correction and repair process of the layered area is completed, and the repaired current frame is obtained.

[0144] Fifth, the correction and repair (also called information recovery) process of the delaminated area.

[0145] Exemplarily, if there is a layered region in the current frame, the layered region of the current frame can be corrected and repaired to make the current frame close to a normal image. For example, information recovery can be performed on the layered region of the current frame based on the current frame. When information recovery is performed on the layered region based on the current frame, a method based on deep learning (neural network generation, etc.) can be used to recover information on the layered region, or other methods can be used to recover information on the layered region. The method based on deep learning is used as an example below.

[0146] See also Figure 5C As shown, it is a schematic diagram of using a deep learning-based method to restore information in a layered area. The process may include: an information recovery network training process, an image frame acquisition process, and a layered area correction and repair process based on the information recovery network. These processes are explained below.

[0147] For the training process of the information recovery network, training data can be obtained, and the information recovery network can be trained based on the training data. For example, based on images of different exposure conditions (layering with different temporal rules, etc.) generated by holmium lasers at different frequencies, these images include normal images without layered areas and layered images with layered areas, and training data can be obtained, and the training data includes historical samples and sample labels.

[0148] A normal image can be used as a sample label, and the layered image before the normal image can be used as a historical sample. Alternatively, a layered image can be used as a historical sample, and the layered image can be repaired to obtain a normal image (there is no restriction on the repair process, such as manual repair, etc.), and the normal image can be used as a sample label. Alternatively, a normal image can be used as a sample label, and a layered area can be added to the normal image to obtain a layered image (i.e., the layered area is superimposed on the normal image to replace a certain area of ​​the normal image), and the layered image can be used as a historical sample. Of course, the above are just a few examples of training data.

[0149] The historical samples (i.e., layered images) and layered region position information (the layered region position information indicates the position of the layered region in the historical samples) in the training data can be input into the model to be trained, and the model to be trained is used to generate correction and repair information of the layered region, and restore information of the layered region through the correction and repair information. Based on this, after the historical samples and the layered region position information are input into the model to be trained, the model to be trained can generate correction and repair information of the layered region based on the historical samples, and the correction and repair information includes the pixel value of each pixel in the layered region. The model to be trained can determine the layered region in the historical sample based on the layered region position information. Then, the model to be trained can use the correction and repair information to restore information of the layered region, that is, for each pixel in the layered region, find the pixel value corresponding to the pixel from the correction and repair information, and assign the pixel value to the pixel, so as to repair each pixel in the layered region. After the above processing, the model to be trained can output a predicted image, which can be an image after the layered region in the historical sample is repaired.

[0150] Then, the similarity between the predicted image and the sample label in the training data can be compared, and the loss value can be calculated based on the similarity between the predicted image and the sample label. Obviously, the greater the similarity, the smaller the loss value, and vice versa, the smaller the similarity, the larger the loss value. Then, the network parameters of the training model can be adjusted based on the loss value, such as by using the gradient descent method, and the adjustment goal is to make the loss value smaller and smaller, and there is no restriction on this process.

[0151] If the adjusted model has converged, the adjusted model is output, that is, the adjusted model is used as the information recovery network. If the adjusted model has not converged, the operation of inputting the historical samples in the training data into the model to be trained (that is, the adjusted model) is returned, and so on, until the adjusted model converges.

[0152] For the image frame acquisition process, the video stream may be acquired, and the current frame may be acquired from the video stream. For example, each image frame in the video stream may be traversed in sequence, and the currently traversed image frame may be used as the current frame.

[0153] For the correction and repair process of the hierarchical area based on the information recovery network, the current frame and the hierarchical area position information of the current frame (determined based on the detection and positioning process of the hierarchical area) can be input into the information recovery network, where the hierarchical area position information can represent the position of the hierarchical area in the current frame.

[0154] The information recovery network can generate correction and repair information of the layered area based on the current frame simulation (for example, the pixel values ​​of the layered area are generated based on the pixel values ​​of the pixels in the non-layered area surrounding the layered area, and there is no limitation on this process), and the correction and repair information may include the pixel value of each pixel in the layered area. The information recovery network can determine the layered area in the current frame based on the position information of the layered area. Then, the information recovery network can use the correction and repair information to perform information recovery on the layered area, that is, for each pixel in the layered area, find the pixel value corresponding to the pixel from the correction and repair information, and assign the pixel value to the pixel, thereby repairing each pixel in the layered area.

[0155] After the above processing, the information recovery network can output the repaired current frame (that is, the layered area is repaired). At this point, the correction and repair process of the layered area is completed to obtain the repaired current frame.

[0156] It can be seen from the above technical solution that if the current frame is an image of the inside of the target object collected by the endoscope, and there is a holmium laser stratification area caused by the holmium laser pulse in the current frame, the information of the holmium laser stratification area is restored. By restoring the holmium laser stratification area, the appearance of the holmium laser stratification area is made close to the normal image, which reduces the visual impairment of the physician, reduces the visual discomfort of the physician, provides a good surgical perception, and achieves the expected surgical effect. Figure 6 The figure shows the real effect of the images before and after holmium laser layering treatment.

[0157] Based on the same application concept as the above method, the present application embodiment proposes an image processing device, see Figure 7 FIG. 1 is a schematic diagram of the structure of the device, wherein the device comprises:

[0158] The determination module 71 is used to determine the layered area of ​​the current frame based on the historical frame before the current frame, the layered area is the information loss area in the current frame, and the information loss area needs to be restored; the processing module 72 is used to restore the information of the layered area of ​​the current frame based on the historical frame; or, restore the information of the layered area of ​​the current frame based on the current frame.

[0159] Exemplarily, when determining the layered area of ​​the current frame based on the historical frame before the current frame, the determination module 71 is specifically used to: slide the current frame to obtain multiple target windows; for each target window, obtain the reference window corresponding to the target window from the historical frame, and determine the similarity between the reference window and the target window; based on the similarity, determine the target window as a layered window or a non-layered window; divide the current frame into multiple areas to be analyzed, and each area to be analyzed includes K consecutive pixel rows, K is a positive integer; for each area to be analyzed, count the number of layered windows in the area to be analyzed; if the number is greater than a number threshold, determine the area to be analyzed as a layered area.

[0160] Exemplarily, when the determination module 71 determines the layered area of ​​the current frame based on the historical frame preceding the current frame, it is specifically used to: input the historical frame to a trained first generator network, and output a simulated layered image through the first generator network, wherein the simulated layered image includes a simulated layered area and a simulated normal area; divide the current frame into a plurality of areas to be analyzed, and for each area to be analyzed, obtain a reference area corresponding to the area to be analyzed from the simulated layered image, and the position of the area to be analyzed in the current frame is the same as the position of the reference area in the simulated layered image; if the reference area is a simulated layered area in the simulated layered image, and the similarity between the area to be analyzed and the simulated layered area is greater than a first similarity threshold, then the area to be analyzed is determined as the layered area of ​​the current frame; if the reference area is a simulated normal area in the simulated layered image, and the similarity between the area to be analyzed and the simulated normal area is less than a second similarity threshold, then the area to be analyzed is determined as the layered area of ​​the current frame.

[0161] Exemplarily, when the determination module 71 determines the hierarchical region of the current frame based on the historical frame preceding the current frame, it is specifically used to: input the historical frame to a trained second generator network, and output a simulated normal image through the second generator network, wherein the simulated normal image includes a simulated normal region; divide the current frame into a plurality of regions to be analyzed, and for each region to be analyzed, obtain a reference region corresponding to the region to be analyzed from the simulated normal image, wherein the position of the region to be analyzed in the current frame is the same as the position of the reference region in the simulated normal image; if the reference region is a simulated normal region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated normal region is less than a similarity threshold, then the region to be analyzed is determined as the hierarchical region of the current frame.

[0162] Exemplarily, when the processing module 72 performs information recovery on the layered area of ​​the current frame based on the historical frame, it is specifically used to: divide the non-layered area of ​​the current frame into multiple image blocks; wherein, for each image block, determine a matching block of the image block from the historical frame, and the matching block is an image block in the historical frame that is most similar to the image block; determine a target motion vector based on the relative displacement between each image block and the matching block of the image block; for each target pixel point in the layered area of ​​the current frame, determine a reference pixel point corresponding to the target pixel point from the historical frame based on the target motion vector; and recover information of the target pixel point based on the pixel value of the reference pixel point.

[0163] Exemplarily, when the processing module 72 performs information recovery on the layered region of the current frame based on the current frame, it is specifically used to: input the current frame and the layered region position information into a trained information recovery network, the layered region position information indicating the position of the layered region in the current frame; simulate and generate correction and repair information through the information recovery network, and determine the layered region based on the layered region position information; wherein the correction and repair information includes the pixel value of each pixel point of the layered region; and perform information recovery on the layered region through the correction and repair information.

[0164] Exemplarily, the device also includes: an acquisition module, used to capture the image of the target object through an endoscope to obtain the historical frame and the current frame, wherein when the current frame is captured by the endoscope, a strong light pulse appears in the environment where the target object is located; wherein the historical frame is an adjacent frame preceding the current frame; or, if there is a stratified area in the adjacent frame preceding the current frame, the historical frame is an adjacent frame after the stratified area is restored by information, or, the historical frame is an adjacent frame before the stratified area is restored by information; or, the historical frame is the first frame preceding the current frame in which no stratified area exists; wherein, when the current frame is captured by the endoscope, the strong light pulse causes the current frame to have a stratified area, the stratified area is a strip-shaped area, the stratified area has a lateral edge, and the brightness of the stratified area is higher than the brightness of the non-stratified area.

[0165] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, see Figure 8 As shown, the electronic device may include: a processor 81 and a machine-readable storage medium 82, the machine-readable storage medium 82 storing machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the image processing method of the above example of the present application.

[0166] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0167] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0168] Based on the same application concept as the above method, the embodiment of the present application also provides a computer program product, including a computer program; when the computer program is executed by a processor, the above image processing method is implemented.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0170] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a hierarchical region of the current frame based on a historical frame preceding the current frame, wherein the hierarchical region is an information loss region in the current frame, and the information loss region needs to be restored; The information of the layered region of the current frame is restored based on the historical frame; or the information of the layered region of the current frame is restored based on the current frame.

2. The method according to claim 1, characterized in that The step of determining the hierarchical region of the current frame based on the historical frames preceding the current frame comprises: Sliding the current frame to obtain a plurality of target windows; for each target window, obtaining a reference window corresponding to the target window from the historical frame, and determining the similarity between the reference window and the target window; determining the target window as a layered window or a non-layered window based on the similarity, wherein the historical frame includes a layered area and / or a non-layered area; The current frame is divided into a plurality of areas to be analyzed, each of which includes K consecutive pixel rows, where K is a positive integer; for each area to be analyzed, the number of layered windows in the area to be analyzed is counted; if the number is greater than a number threshold, the area to be analyzed is determined as a layered area.

3. The method according to claim 1, characterized in that: The step of determining the hierarchical region of the current frame based on the historical frames preceding the current frame comprises: Inputting the historical frame to a trained first generator network, and outputting a simulated layered image through the first generator network, wherein the simulated layered image includes a simulated layered area and a simulated normal area; Dividing the current frame into a plurality of regions to be analyzed, and for each region to be analyzed, obtaining a reference region corresponding to the region to be analyzed from the simulated layered image, wherein a position of the region to be analyzed in the current frame is the same as a position of the reference region in the simulated layered image; If the reference region is a simulated layered region in the simulated layered image, and the similarity between the region to be analyzed and the simulated layered region is greater than a first similarity threshold, then determining the region to be analyzed as the layered region of the current frame; If the reference area is a simulated normal area in the simulated layered image, and the similarity between the area to be analyzed and the simulated normal area is less than a second similarity threshold, the area to be analyzed is determined as the layered area of ​​the current frame.

4. The method according to claim 1, characterized in that: The step of determining the hierarchical region of the current frame based on the historical frames preceding the current frame comprises: Inputting the historical frame to a trained second generator network, and outputting a simulated normal image through the second generator network, wherein the simulated normal image includes a simulated normal area; Dividing the current frame into a plurality of regions to be analyzed, and for each region to be analyzed, acquiring a reference region corresponding to the region to be analyzed from the simulated normal image, wherein a position of the region to be analyzed in the current frame is the same as a position of the reference region in the simulated normal image; If the reference region is a simulated normal region in the simulated layered image, and the similarity between the region to be analyzed and the simulated normal region is less than a similarity threshold, the region to be analyzed is determined as the layered region of the current frame.

5. The method according to claim 1, characterized in that The restoring information of the layered area of ​​the current frame based on the historical frame includes: Divide the non-layered area of ​​the current frame into a plurality of image blocks; for each image block, determine a matching block of the image block from the historical frame, the matching block being an image block in the historical frame that is most similar to the image block; determine a target motion vector based on a relative displacement between each image block and the matching block of the image block; For each target pixel point in the hierarchical area of ​​the current frame, determining a reference pixel point corresponding to the target pixel point from the historical frame based on the target motion vector; The information of the target pixel point is restored based on the pixel value of the reference pixel point.

6. The method according to claim 1, characterized in that The recovering information of the layered area of ​​the current frame based on the current frame includes: Inputting the current frame and the hierarchical region position information into a trained information recovery network, wherein the hierarchical region position information indicates the position of the hierarchical region in the current frame, and the information recovery network is used to recover information of the hierarchical region; Correction and repair information is generated through the information recovery network simulation, and the layered area is determined based on the layered area position information; wherein the correction and repair information includes the pixel value of each pixel point of the layered area; and information of the layered area is restored through the correction and repair information.

7. The method according to any one of claims 1 to 6, characterized in that: Before determining the hierarchical region of the current frame based on the historical frames preceding the current frame, the method further includes: Acquiring an image of the target object through an endoscope to obtain the historical frame and the current frame, wherein when the current frame is acquired through the endoscope, a strong light pulse appears in the environment where the target object is located; The historical frame is an adjacent frame preceding the current frame; or, if the adjacent frame preceding the current frame has a layered region, the historical frame is an adjacent frame after the layered region is restored by information; or, the historical frame is an adjacent frame before the layered region is restored by information; or, the historical frame is the first frame preceding the current frame that does not have a layered region; When the current frame is collected through the endoscope, the strong light pulse causes a stratified area to exist in the current frame. The stratified area is a strip-shaped area with a lateral edge, and the brightness of the stratified area is higher than that of a non-stratified area.

8. An image processing device, characterized in that: The device comprises: A determination module, configured to determine a hierarchical region of the current frame based on a historical frame preceding the current frame, wherein the hierarchical region is an information loss region in the current frame and information recovery is required for the information loss region; A processing module is used to restore information of the layered area of ​​the current frame based on the historical frame; or to restore information of the layered area of ​​the current frame based on the current frame.

9. The device according to claim 8, It is characterized in that in, When the determination module determines the layered area of ​​the current frame based on the historical frame before the current frame, it is specifically used to: perform a sliding window operation on the current frame to obtain multiple target windows; for each target window, obtain a reference window corresponding to the target window from the historical frame, and determine the similarity between the reference window and the target window; determine the target window as a layered window or a non-layered window based on the similarity; divide the current frame into multiple areas to be analyzed, and each area to be analyzed includes K consecutive pixel rows, K is a positive integer; for each area to be analyzed, count the number of layered windows in the area to be analyzed; if the number is greater than a number threshold, determine the area to be analyzed as a layered area; Wherein, when the determination module determines the hierarchical region of the current frame based on the historical frame preceding the current frame, it is specifically used to: input the historical frame to the trained first generator network, and output a simulated hierarchical image through the first generator network, wherein the simulated hierarchical image includes a simulated hierarchical region and a simulated normal region; divide the current frame into a plurality of regions to be analyzed, and for each region to be analyzed, obtain a reference region corresponding to the region to be analyzed from the simulated hierarchical image, wherein the position of the region to be analyzed in the current frame is the same as the position of the reference region in the simulated hierarchical image; if the reference region is a simulated hierarchical region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated hierarchical region is greater than a first similarity threshold, then determine the region to be analyzed as the hierarchical region of the current frame; if the reference region is a simulated normal region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated normal region is less than a second similarity threshold, then determine the region to be analyzed as the hierarchical region of the current frame; Wherein, when the determination module determines the hierarchical region of the current frame based on the historical frame preceding the current frame, it is specifically used to: input the historical frame to the trained second generator network, and output a simulated normal image through the second generator network, wherein the simulated normal image includes a simulated normal region; divide the current frame into a plurality of regions to be analyzed, and for each region to be analyzed, obtain a reference region corresponding to the region to be analyzed from the simulated normal image, wherein the position of the region to be analyzed in the current frame is the same as the position of the reference region in the simulated normal image; if the reference region is a simulated normal region in the simulated hierarchical image, and the similarity between the region to be analyzed and the simulated normal region is less than a similarity threshold, then determine the region to be analyzed as the hierarchical region of the current frame; Wherein, when the processing module performs information recovery on the hierarchical area of ​​the current frame based on the historical frame, it is specifically used to: divide the non-hierarchical area of ​​the current frame into multiple image blocks; wherein, for each image block, determine a matching block of the image block from the historical frame, and the matching block is the image block in the historical frame that is most similar to the image block; determine a target motion vector based on the relative displacement between each image block and the matching block of the image block; for each target pixel point in the hierarchical area of ​​the current frame, determine a reference pixel point corresponding to the target pixel point from the historical frame based on the target motion vector; and perform information recovery on the target pixel point based on the pixel value of the reference pixel point; Wherein, when the processing module performs information restoration on the hierarchical region of the current frame based on the current frame, it is specifically used to: input the current frame and the hierarchical region position information into the trained information restoration network, wherein the hierarchical region position information indicates the position of the hierarchical region in the current frame; simulate and generate correction and restoration information through the information restoration network, and determine the hierarchical region based on the hierarchical region position information; wherein the correction and restoration information includes the pixel value of each pixel point of the hierarchical region; and perform information restoration on the hierarchical region through the correction and restoration information; The device further includes: an acquisition module, which is used to capture the image of the target object through an endoscope to obtain the historical frame and the current frame, wherein when the current frame is captured through the endoscope, a strong light pulse appears in the environment where the target object is located; wherein the historical frame is an adjacent frame preceding the current frame; or, if there is a stratified area in the adjacent frame preceding the current frame, the historical frame is an adjacent frame after the stratified area is restored by information, or, the historical frame is an adjacent frame before the stratified area is restored by information; or, the historical frame is the first frame preceding the current frame in which no stratified area exists; wherein, when the current frame is captured through the endoscope, the strong light pulse causes the current frame to have a stratified area, the stratified area is a strip-shaped area, the stratified area has a lateral edge, and the brightness of the stratified area is higher than that of the non-stratified area.

10. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-7.

Citation Information

Patent Citations

  • Video restoration method and device

    CN115511721A

  • Image processing method, system and device for eliminating laser layering

    CN115988325A

  • Image processing method, system and device for eliminating laser layering

    CN116033273A

  • Image correction method, device and equipment

    CN117132489A

  • Image block recovery method and device, computer equipment and storage medium

    CN117830441A