A neuroendoscopic image defogging method, device and system

Abstract

The present application relates to the field of image processing technology, and specifically to a method, device and system for defogging an image of a neuroendoscope, which specifically includes: constructing a moving distance feature value by analyzing the distribution of surgical instruments and human tissues in a surgical area in a grayscale image of a neuroendoscope, and constructing a target suspicion degree in combination with the color consistency degree and the contour symmetry degree of the connected domain; constructing a template block side length by the target suspicion degree, and obtaining a reflective area mask of the neuroendoscopic image by threshold segmentation of an HSV image of the neuroendoscopic image; repairing the reflective area in the neuroendoscopic image based on the template block side length and the reflective area mask in combination with the Criminisi algorithm, and defogging the repaired neuroendoscopic image by using a convolutional neural network; improving the repair effect of the reflective area in the neuroendoscopic image, thereby improving the quality of the neuroendoscopic image, and thus improving the defogging effect of the neuroendoscopic image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and system for defogging images of a neuroendoscopy. Background Art

[0002] During neuroendoscopic surgery, due to the high humidity inside the human body, fog is easily generated on the lens surface of the neuroendoscopic camera, which will cause the contrast and clarity of the collected neuroendoscopic images to decrease. In order to improve the clarity of the scene in the surgical field, the collected neuroendoscopic images need to be defogged. However, when using a neural network to extract the feature map of the original neuroendoscopic image, it is usually necessary to repair the reflective area in the original neuroendoscopic image. This is because the reflective area in the original neuroendoscopic image is usually caused by the human tissue with a moist and smooth surface in the surgical area and the surgical instruments with reflective properties reflecting the light source. If the neuroendoscopic image with the reflective area is directly defogged, the neural network will not be able to fully reflect the real surgical scene characteristics in the neuroendoscopic image when extracting the feature map, which will cause the doctor to be unable to effectively obtain the real situation of the surgical area from the defogged image of the neuroendoscopic image, thereby interfering with the doctor's observation and diagnosis.

[0003] The Criminisi algorithm is a commonly used image restoration algorithm. However, in the traditional Criminisi algorithm, the size of the template block is usually fixed. In neuroendoscopic images, the size and shape of the reflective area will vary greatly depending on the surgical area. If the size of the template block is set unreasonably, the detailed information of the human tissue in the reflective area cannot be effectively restored. At the same time, the unreasonable template block size will also increase the restoration time of the reflective area caused by the surgical instrument, thereby affecting the final restoration effect and restoration time of the reflective area in the collected neuroendoscopic image, thereby affecting the defogging effect. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide a method, device and system for defogging images in a neuroendoscopy. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for defogging an image in a neuroendoscopy, the method comprising the following steps:

[0006] Collect each frame of neuroendoscopic image of the surgical area; perform grayscale image and HSV image conversion respectively;

[0007] Obtain each connected domain in each grayscale image; obtain the optical flow vector of each pixel in each frame of neuroendoscopic image through the optical flow estimation algorithm as the optical flow vector of each pixel in the corresponding grayscale image; construct the moving distance feature value of each connected domain based on the overall size and discreteness of the optical flow vectors of all pixels in each connected domain; construct the target suspicion degree of each connected domain by combining the similarity of the chromaticity values ​​of the pixels in the HSV image corresponding to each connected domain and the degree of confusion of the gradient direction angle of all edge pixels in each connected domain;

[0008] Determine the target suspicion degree of each pixel in each grayscale image based on the target suspicion degree; perform threshold segmentation on the brightness values ​​of all pixels in each HSV image to obtain each binary image; construct the template block size of each pixel in each frame of neuroendoscopic image based on the target suspicion degree of each pixel;

[0009] Each binary image is used as a mask for the reflective area of ​​each frame of the neuroendoscopic image, and the reflective area in the neuroendoscopic image is repaired using an image repair algorithm in combination with the template block size; an image sequence consisting of repaired images of all frames of the neuroendoscopic image is obtained, and the image sequence is defogged using a neural network.

[0010] In one embodiment, the calculation expression of the moving distance characteristic value of each connected domain is:

[0011] , where Represents a connected domain The moving distance characteristic value, Represents a connected domain The mean value of the modulus of the optical flow vector of all pixels in ; Represents a connected domain The standard deviation of the optical flow vector modulus of all pixels in Represents the jth connected component of the i-th grayscale image.

[0012] In one embodiment, the process of obtaining the target suspicion degree of each connected domain is as follows:

[0013] The expression for constructing the color consistency of each connected domain is: , where Connected domain The color consistency of The standard deviation of the chromaticity values ​​of all pixels corresponding to all pixels in the i-th frame HSV image; Indicates an artificially preset positive number;

[0014] The contour symmetry degree of each connected domain is constructed based on the chaos degree of the gradient direction angle of all edge pixels in each connected domain;

[0015] The fusion value of the moving distance feature value, color consistency and contour symmetry of each connected domain is used as the target suspicion of each connected domain.

[0016] In one embodiment, the process of obtaining the degree of contour symmetry is as follows:

[0017] The gradient direction angle of each edge pixel point in each connected domain is calculated by using a gradient detection algorithm; all the gradient direction angles of each connected domain are statistically histogrammed to obtain each histogram; The degree of symmetry of the outline is recorded as , The calculation expression is: , where p is the connected domain The standard deviation of the height values ​​of all bins in the histogram; a is an artificially preset positive number.

[0018] In one embodiment, the target suspicion degree of each pixel is the target suspicion degree of the connected domain where the pixel is located.

[0019] In one embodiment, the process of obtaining the template block size of each pixel point is as follows:

[0020] The side length of the template block of the u-th pixel in the i-th frame of neuroendoscopic image is recorded as , The calculation expression is: , where f1 and f2 represent the template block size parameters; Indicates the target suspicion of the u-th pixel in the i-th grayscale image; To find the function that is closest to an odd integer;

[0021] The size of the template block of each pixel is obtained by the side length of the template block of each pixel.

[0022] In one embodiment, the process of acquiring the repaired image is as follows:

[0023] Each frame of neuroendoscopic image and the mask of the corresponding reflective area are used as the input of the Criminisi algorithm, wherein the template block size of each pixel point in each frame of neuroendoscopic image is used as the template block size of the pixel point in the Criminisi algorithm, and the repaired image of the neuroendoscopic image is output.

[0024] In one embodiment, defogging the image sequence by using a neural network includes:

[0025] Based on the first convolutional neural network, the original imaging feature map of all neuroendoscopic images in the image sequence is extracted; based on the second convolutional neural network, the smoke feature image sequence of the image sequence is extracted, and based on the smoke feature image sequence, the final smoke feature map of the corresponding neuroendoscopic image is generated; based on the original imaging feature map and the final smoke feature map, the neuroendoscopic image after defogging is obtained.

[0026] In a second aspect, the present application also provides an image defogging device for a neuroendoscopy, comprising:

[0027] Image acquisition module: collects each frame of neuroendoscopic images of the surgical area; performs grayscale image and HSV image conversion respectively;

[0028] Image analysis module: obtain each connected domain in each grayscale image; obtain the optical flow vector of each pixel in each frame of neuroendoscopic image through the optical flow estimation algorithm as the optical flow vector of each pixel in the corresponding grayscale image; construct the moving distance feature value of each connected domain based on the overall size and discreteness of the optical flow vectors of all pixels in each connected domain; combine the similarity of the chromaticity values ​​of the pixels in the HSV image corresponding to each connected domain, and the degree of confusion of the gradient direction angle of all edge pixels in each connected domain, to construct the target suspicion of each connected domain;

[0029] Template block size calculation module: determine the target suspicion of each pixel in each grayscale image based on the target suspicion; perform threshold segmentation on the brightness values ​​of all pixels in each HSV image to obtain each binary image; construct the template block size of each pixel in each frame of neuroendoscopic image based on the target suspicion of each pixel;

[0030] Image defogging module: each binary image is used as a mask for the reflective area of ​​each frame of the neuroendoscopic image, and the reflective area in the neuroendoscopic image is repaired using an image restoration algorithm in combination with the template block size; an image sequence consisting of the restored images of all frames of the neuroendoscopic image is obtained, and the image sequence is defogged through a neural network.

[0031] In a third aspect, an embodiment of the present application also provides an image defogging system for a neuroendoscopy, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0032] The embodiments of the present application have at least the following beneficial effects:

[0033] The present application analyzes the distribution of surgical instruments and human tissues in the surgical area in the grayscale image of the neuroendoscope, constructs a moving distance feature value in combination with the connected domain and optical flow vector of the grayscale image, and constructs a target suspicion degree in combination with the color consistency degree and the contour symmetry degree of the connected domain. The beneficial effect of the application is that the difference in moving characteristics between surgical instruments and human tissues in the surgical area, the color difference between human tissues and surgical instruments in the surgical area, and the difference in edge contour symmetry between human tissues and surgical instruments are taken into account, thereby improving the distinction between the corresponding image areas of surgical instruments and human tissues in the surgical area in the grayscale image of the neuroendoscope.

[0034] The present application constructs the side length of the template block through the target suspicion degree, obtains the reflective area mask of the neuroendoscopic image through the threshold segmentation of the HSV image of the neuroendoscopic image; based on the side length of the template block and the reflective area mask, the reflective area in the neuroendoscopic image is repaired in combination with the Criminisi algorithm, and the image sequence obtained based on the repaired image of the neuroendoscopic image is used to achieve the defogging of the repaired neuroendoscopic image using the image defogging method based on the convolutional neural network. The beneficial effect is that the template block size of the pixel points in the reflective area of ​​the neuroendoscopic image is reasonably set in the Criminisi algorithm, while improving the repair effect of the human tissue detail information in the reflective area of ​​the neuroendoscopic image, the repair time of the reflective area caused by the surgical instrument in the neuroendoscopic image is reduced, and the reflective area of ​​the neuroendoscopic image can be effectively repaired in time, so as to more accurately and timely reflect the real situation of the surgical area in the neuroendoscopic image after defogging, thereby improving the defogging effect of the neuroendoscopic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flowchart of a method for defogging an image of a neuroendoscopy provided in one embodiment of the present application;

[0037] Figure 2 is the grayscale image of the neuroendoscopic image;

[0038] Figure 3 It is the binary image of HSV image;

[0039] Figure 4This is a schematic diagram of the structure of an image defogging device for a neuroendoscopy. DETAILED DESCRIPTION

[0040] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the image defogging method, device and system of a neuroendoscopy proposed in the present application, its specific implementation, structure, features and effects. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0041] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0042] The following is a detailed description of a specific scheme of a neuroendoscopic image defogging method, device and system provided by the present application in conjunction with the accompanying drawings.

[0043] See also Figure 1 , which shows a flowchart of a method for defogging an image of a neuroendoscopy provided by an embodiment of the present application, the method comprising the following steps:

[0044] Step S1, acquiring each frame of neuroendoscopic image of the surgical area, and converting the image into grayscale image and HSV image respectively.

[0045] The present application aims to repair the reflective area when the neural network extracts the feature map of the neuroendoscopic image through the Criminisi algorithm, so as to improve the accuracy of the neural network in extracting the real detail features in the human tissue area in the neuroendoscopic image.

[0046] During the process of performing surgery on a patient using a neuroendoscopy, a camera of the neuroendoscopy is used to collect continuous T frame images in the patient's surgical area. Preferably, in one embodiment of the present application, the value of T is set to 10. As other embodiments, the value of T can be set by the implementer according to actual conditions.

[0047] All collected images are denoised using bilateral filtering to reduce the impact of noise generated during image acquisition and transmission, and obtain T-frame neuroendoscopic images after denoising. Bilateral filtering is a well-known technology, and the specific process is not repeated here. It should be noted that for denoising all images, this application only provides a denoising method. There are many existing denoising methods, and implementers can also use other denoising methods to denoise images, and this application does not make specific restrictions.

[0048] Each frame of the obtained neuroendoscopic image is converted into a grayscale image and an HSV image to obtain each frame of grayscale image and each frame of HSV image. Among them, the conversion of RGB image into grayscale image and HSV image is a well-known technology, and the specific process is not repeated here.

[0049] Step S2, based on the movement characteristics of the pixels between each frame and its previous frame of neuroendoscopic image, construct the movement distance characteristic value of each connected domain in the corresponding grayscale image; combined with the similarity of the chromaticity values ​​of the pixels in the HSV image corresponding to each connected domain, and the degree of confusion of the gradient direction angles of all edge pixels in each connected domain, construct the target suspicion degree of each connected domain.

[0050] (1) Get each connected domain in each grayscale image:

[0051] The i-th frame grayscale image is recorded as , as a grayscale image For example, the edge image of the grayscale image is extracted using the Canny edge detection algorithm. The Canny edge detection algorithm is a well-known technology, and the specific process is not repeated here. It should be noted that there are many existing edge detection methods for obtaining the edge image of the grayscale image, and the implementer may also use other edge detection algorithms to obtain the edge image of the grayscale image, and this application does not make specific restrictions.

[0052] By extracting all connected domains in the edge image, all connected domains of the grayscale image are obtained, and each connected domain represents each image region corresponding to the surgical area in the grayscale image.

[0053] (2) Each frame of neuroendoscopic image and its previous frame are input into the optical flow estimation algorithm, and the optical flow vector of each pixel in each frame of neuroendoscopic image is output as the optical flow vector of each pixel in the corresponding grayscale image; based on the overall size and discreteness of the optical flow vectors of all pixels in each connected domain, the movement characteristics between the corresponding areas in two adjacent frames of neuroendoscopic image are analyzed, and the moving distance characteristic value of each connected domain is constructed:

[0054] In neuroendoscopic surgery, in order to ensure the stability and accuracy of the surgery, the camera of the neuroendoscopic camera usually needs to remain stable and avoid large-scale movement, so that the human tissue area in the adjacent frame images collected will not move in a large range, and the movement range of each position in the human tissue area that is not in contact with the surgical instrument is relatively consistent, while the surgical instrument needs to operate in a large range in the surgical area to perform various surgical tasks, such as resection, electrocoagulation, suction, etc. This causes the surgical instrument area in the adjacent frame images collected to move in a large range, and the movement range of each position in the surgical instrument area is usually inconsistent. These phenomena will cause the neural network to extract deviations or inaccuracies in the feature images when processing neuroendoscopic images, such as misjudging the image features of the surgical instrument area in the neuroendoscopic image as image features in the human tissue area.

[0055] (2.1) The i-th frame neuroendoscopic image and the (i-1)-th frame neuroendoscopic image are used as inputs of the Farneback optical flow estimation algorithm, and the output is the optical flow vector of each pixel in the i-th frame neuroendoscopic image, which is used as the optical flow vector of each pixel corresponding to the i-th frame grayscale image, and is used to characterize the motion displacement vector between a certain position in the surgical area corresponding to each pixel in the i-th frame grayscale image and the pixel corresponding to the position in the (i-1)-th frame grayscale image. The Farneback optical flow algorithm is a well-known technology, and the specific process is not repeated here.

[0056] It should be noted that for the calculation of the optical flow vector of each pixel point in the neuroendoscopic image, the present application only provides an optical flow estimation algorithm. There are many existing optical flow estimation algorithms, and implementers may also use other optical flow estimation algorithms to calculate the optical flow vector of each pixel point in the neuroendoscopic image. The present application does not make any specific restrictions.

[0057] (2.2) The jth connected domain in the i-th grayscale image is denoted as , with connected domain As an example, the moving distance feature value of the connected domain is constructed based on the overall size and discreteness of the optical flow vectors of all pixels in the connected domain to characterize the connected domain. The possibility that the connected domain corresponding to the surgical instrument has the movement characteristics in the i-th frame and the (i-1)-th frame of the neuroendoscopic image, the connected domain The expression of the moving distance characteristic value is:

[0058] , where Represents a connected domain The moving distance characteristic value, Represents a connected domain The mean of the optical flow vector modulus of all pixels in ; Represents a connected domain The standard deviation of the optical flow vector norm of all pixels in .

[0059] Connected Domain The larger the movement range of the corresponding surgical area between the (i-1)th frame and the i-th frame of the neuroendoscopic image, that is, The larger the connected domain The more inconsistent the movement range of each position in the corresponding surgical area between the (i-1)th frame and the i-th frame of the neuroendoscopic image, the more the connected domain The more inconsistent the optical flow vector modulus of each pixel in the image is, the greater the degree of discreteness is. The larger the value, the more connected the domain is. The more similar the movement characteristics between the (i-1)th and i-th neuroendoscopic images are to the movement characteristics of the surgical instrument between two adjacent neuroendoscopic images, the more connected the domain is. The greater the possibility that the surgical instrument is located in the connected domain, that is, the larger the moving distance eigenvalue, the more likely the connected domain is. The more likely it is that the surgical instrument area in the surgical area corresponds to the connected domain in the i-th frame grayscale image.

[0060] (3) Calculate the color consistency of each connected domain:

[0061] However, in neuroendoscopic surgery, various locations in the area of ​​human tissue in contact with surgical instruments will also move to varying degrees due to the operation of the surgical instruments, and these movement ranges are usually inconsistent. For example, operations such as resection, electrocoagulation, and suction often require pulling the human tissue in contact with the surgical instruments in different directions and depths. However, surgical instruments are usually made of metal or alloys, so that they have a consistent metal color, and the design of surgical instruments is usually symmetrical to ensure stability during surgery. For example, the cylindrical suction head, the ring-shaped scraper head, and the conical bipolar electrocoagulation forceps all have symmetrical contour edges, while the color and shape of human tissue in the surgical area are diverse because they are composed of different cell types and structures.

[0062] Based on the above analysis, the connected domain For example, the color consistency of the connected domain is constructed based on the similarity of the chromaticity values ​​of the pixels in the HSV image corresponding to the connected domain, which is used to characterize the consistency of the color distribution in the surgical area corresponding to the connected domain. The calculation expression of the color consistency of the connected domain is:

[0063] , where Connected domain The color consistency of The standard deviation of the chromaticity values ​​of all pixels corresponding to all pixels in the i-th frame HSV image; Indicates a positive number that is artificially preset to prevent the denominator from being 0. Preferably, in one embodiment of the present application, The value of is set to 0.01. As another embodiment, The implementer can set the value according to the actual situation.

[0064] Connected Domain The closer the colors of the corresponding positions in the surgical area are, that is, the smaller w is, the more connected the domain is. The greater the consistency of the color distribution in the corresponding surgical area, the more connected the domain is. The more likely it is that the surgical instrument area in the surgical area corresponds to the connected domain in the i-th frame grayscale image.

[0065] (4) Furthermore, based on the distribution of the gradient direction angles of all edge pixels in each connected domain, the degree of disorder is analyzed and the degree of symmetry of the contour of each connected domain is constructed:

[0066] (4.1) Use the Sobel operator to calculate the connected domains The gradient direction angle of each edge pixel in the connected domain All the gradient direction angles are histogram-statisticed to obtain a histogram of the gradient direction angles, which is used to characterize the connected domain. The distribution of the contour edge of the corresponding surgical area. The Sobel operator and histogram statistics are both well-known technologies, and the specific process will not be repeated here.

[0067] It should be noted that for the calculation of the gradient direction angle of edge pixels, this application only provides a gradient detection method. There are many existing gradient detection methods, and implementers can also use other gradient detection algorithms to calculate the gradient direction angle of edge pixels. This application does not make specific restrictions.

[0068] (4.2) Connected domain The calculation expression of the contour symmetry degree is: , where Connected domain The degree of symmetry of the outline; p is the connected domain The standard deviation of the height values ​​of all bins in the histogram; a is a preset positive number to prevent the denominator from being 0. In the embodiment of the present application, the value of a is set to 0.01. As other embodiments, the value of a can be set by the implementer according to the actual situation.

[0069] Connected Domain The degree of outline symmetry is used to characterize the connected domain The symmetry of the contour edge of the corresponding surgical area. Since the symmetrical contour has similar edge distribution in multiple directions of the contour edge, the more uniform the distribution of the gradient direction angle of the edge pixel points in the histogram, the more connected the domain is. The more obvious the symmetry of the contour edge of the corresponding surgical area, the higher the degree of contour symmetry. The larger the connected domain The more likely it is that the surgical instrument area in the surgical area corresponds to the connected domain in the i-th frame grayscale image.

[0070] (5) Based on the above analysis, the target suspicion of each connected domain is constructed: the fusion value of the moving distance feature value, color consistency and contour symmetry of each connected domain is used as the target suspicion of each connected domain to characterize the possibility that the surgical area corresponding to each connected domain is the area where the surgical instrument is located.

[0071] It should be noted that the fusion described in this application is to combine multiple variables. The specific fusion method can be determined according to actual conditions during the application process, and this application does not impose any special restrictions.

[0072] Preferably, in one embodiment of the present application, the calculation expression of the target suspicion degree can be: , where Connected domain The target suspicion, , and Represent connected domains The moving distance characteristic value, color consistency, and contour symmetry; is the normalization function.

[0073] In other embodiments of the present application, the calculation expression of the target suspicion degree may be: .

[0074] Connected Domain The more the connected domain corresponding to the surgical instrument has the moving characteristics between the (i-1)th frame and the i-th frame of the neuroendoscopic image, that is, The larger the connected domain The more consistent the color distribution in the corresponding surgical area, the higher the symmetry of the edge contour of the surgical area, that is, and The larger the value, the more connected the domain is. The greater the possibility that the corresponding surgical area is the area where the surgical instrument is located, that is, the higher the target suspicion The bigger.

[0075] Step S3, based on the target suspicion degree, determine the target suspicion degree of each pixel in each grayscale image; perform threshold segmentation on the brightness values ​​of all pixels in each HSV image to obtain each binary image; and construct a template block size of each pixel in each frame of neuroendoscopic image based on the target suspicion degree of the pixel.

[0076] (1) Taking pixel k in the grayscale image of the i-th frame as an example, if pixel k is in the connected domain , then the connected domain The target suspicion degree of pixel k is used as the target suspicion degree of pixel k, otherwise the target suspicion degree of pixel k is assigned to 0. The purpose of this is to improve the subsequent repair effect of pixels in the reflective area of ​​the neuroendoscopic image containing human tissue, so that the feature map extracted by the neural network can fully reflect the detailed information of the human tissue in the neuroendoscopic image.

[0077] (2) Since the reflective area in the neuroendoscopic image has a larger brightness value than the normal area, for the grayscale image and the corresponding HSV image of each frame of the neuroendoscopic image, the brightness values ​​of all pixels in the HSV image corresponding to the grayscale image are used as the input of the maximum inter-class variance algorithm, and the segmentation threshold v is output. The maximum inter-class variance algorithm is a well-known technology, and the specific process is not repeated here.

[0078] It should be noted that for calculating the segmentation threshold of the brightness values ​​of all pixels in the HSV image, this application only provides a threshold segmentation algorithm. There are many existing threshold segmentation methods. Implementers can also use other threshold segmentation algorithms to calculate the segmentation threshold of the brightness values ​​of all pixels in the HSV image. This application does not make specific restrictions.

[0079] (3) Taking the i-th frame grayscale image as an example, the grayscale image is Figure 2 As shown in , the pixels whose brightness values ​​in the HSV image corresponding to the i-th frame grayscale image are greater than the segmentation threshold v are assigned a value of 1, and the pixels whose brightness values ​​are less than or equal to the segmentation threshold v are assigned a value of 0, and a binary image is obtained, as shown in Figure 3 As shown. Since the pixel points with a pixel value of 1 in the binary image represent the pixel points in the reflective area, the binary image is used as a mask of the reflective area of ​​the i-th frame of the neuroendoscopic image to mark the pixel points in the reflective area that needs to be repaired in the neuroendoscopic image.

[0080] (4) Furthermore, taking the i-th frame of neuroendoscopic image as an example, the template block side length of each pixel in the neuroendoscopic image is constructed based on the target suspicion of each pixel in the corresponding grayscale image, which is used to determine the template block side length of each pixel when the reflective area in the neuroendoscopic image is repaired using the Criminisi algorithm. The calculation expression of the template block side length of each pixel in the neuroendoscopic image is:

[0081] , where is the side length of the template block at the u-th pixel in the i-th frame of the neuroendoscopic image; f1 and f2 represent the template block size parameters, which are used to control the size of the template block. Both f1 and f2 are odd numbers; Indicates the target suspicion of the u-th pixel in the i-th frame grayscale image; To find the function of the nearest odd number. Preferably, in one embodiment of the present application, the values ​​of f1 and f2 are set to 3 and 9 respectively. As other embodiments, the values ​​of f1 and f2 can be set by the implementer according to the actual situation.

[0082] During surgery, the details of human tissue in the surgical area are more important than the details of the surgical instrument surface. Therefore, the greater the possibility that the pixel corresponding to the u-th pixel in the i-th frame of the neuroendoscopic image in the i-th frame of the grayscale image is the pixel corresponding to the surgical instrument area in the surgical area in the i-th frame of the grayscale image, that is, The larger the size, the larger the template block size of the u-th pixel in the i-th frame of the neuroendoscopic image should be to reduce the repair time in the reflective area where the pixel is located, that is, the template block side length The bigger; The smaller it is, the greater the possibility that the u-th pixel in the i-th frame neuroendoscopic image is the pixel corresponding to the human tissue area in the surgical area, and the smaller the template block size of the pixel should be, the more effective it is to repair the detailed information of the human tissue in the reflective area where the pixel is located. That is, the side length of the template block The smaller.

[0083] By reasonably setting the template block size of the pixels in the reflective area of ​​the neuroendoscopic image in the Criminisi algorithm, the detailed information of the human tissue in the reflective area of ​​the neuroendoscopic image can be more accurately restored, which is conducive to making the feature map extracted by the neural network more fully reflect the real surgical scene characteristics in the neuroendoscopic image.

[0084] Step S4, based on the side length of the template block and the masks of each reflective area, the reflective area in each neuroendoscopic image is repaired using the Criminisi algorithm; based on an image sequence composed of repaired images of all frames of neuroendoscopic images, the neuroendoscopic images are defogged using a convolutional neural network.

[0085] The i-th frame of neuroendoscopic image and the corresponding reflective area mask are used as the input of the Criminisi algorithm, wherein the template block size of the pixel point obtained based on the template block side length of each pixel point in the i-th frame of neuroendoscopic image is used as the size of the template block size of the pixel point in the Criminisi algorithm, and the repaired image B(i) of the i-th frame of neuroendoscopic image is output. For example, if the template block side length of the u-th pixel point in the i-th frame of neuroendoscopic image is 3, then the template block size of the u-th pixel point in the i-th frame of neuroendoscopic image in the Criminisi algorithm is Among them, the Criminisi algorithm is a well-known technology, and the specific process will not be repeated here.

[0086] Based on each frame of the neuroendoscopic image, the same acquisition method as that of the repaired image B(i) is adopted to obtain the repaired image of each frame of the neuroendoscopic image. All the repaired images are arranged in ascending order according to the frame number of the corresponding neuroendoscopic image. The obtained image sequence is recorded as the image sequence C to be processed. The image sequence C to be processed includes all the neuroendoscopic images that have been repaired in the reflective area, and each image in the image sequence to be processed has a different contrast.

[0087] The above image sequence to be processed is dehazed by a convolutional neural network, including:

[0088] Based on the first convolutional neural network, the original imaging feature map of all neuroendoscopic images in the image sequence C to be processed is extracted; based on the second convolutional neural network, the smoke feature image sequence of the image sequence C to be processed is extracted, and the final smoke feature map of the corresponding neuroendoscopic image is generated based on the smoke feature image sequence; based on the original imaging feature map and the final smoke feature map, the imaging feature map after defogging is obtained to achieve defogging of the neuroendoscopic image. Among them, the first and second convolutional neural networks are both trained convolutional neural networks, and the training process is a well-known technology, and the specific process will not be repeated. Preferably, in one embodiment of the present application, the first and second convolutional neural networks are both ResNet-50 convolutional neural networks (which have Convolution kernel). In other embodiments of the present application, the first and second convolutional neural networks can be set by the implementer, and the present application does not impose any specific restrictions. Among them, the steps of defogging the image sequence to be processed by the convolutional neural network are well-known technologies, and the specific process will not be repeated.

[0089] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a neuroendoscopic image defogging device provided in an embodiment of the present application. In this embodiment, each unit included in the terminal is used to execute each step in an embodiment corresponding to a neuroendoscopic image defogging method. Figure 4 , the image defogging device of the neuroendoscopy includes:

[0090] Image acquisition module: collects each frame of neuroendoscopic images of the surgical area; performs grayscale image and HSV image conversion respectively;

[0091] Image analysis module: obtain each connected domain in each grayscale image; obtain the optical flow vector of each pixel in each frame of neuroendoscopic image through the optical flow estimation algorithm as the optical flow vector of each pixel in the corresponding grayscale image; construct the moving distance feature value of each connected domain based on the overall size and discreteness of the optical flow vectors of all pixels in each connected domain; combine the similarity of the chromaticity values ​​of the pixels in the HSV image corresponding to each connected domain, and the degree of confusion of the gradient direction angle of all edge pixels in each connected domain, to construct the target suspicion of each connected domain;

[0092] Template block size calculation module: determine the target suspicion of each pixel in each grayscale image based on the target suspicion; perform threshold segmentation on the brightness values ​​of all pixels in each HSV image to obtain each binary image; construct the template block size of each pixel in each frame of neuroendoscopic image based on the target suspicion of each pixel;

[0093] Image defogging module: each binary image is used as a mask for the reflective area of ​​each frame of the neuroendoscopic image, and the reflective area in the neuroendoscopic image is repaired using an image restoration algorithm in combination with the template block size; an image sequence consisting of the restored images of all frames of the neuroendoscopic image is obtained, and the image sequence is defogged through a neural network.

[0094] Based on the same inventive concept as the above method, an embodiment of the present application also provides an image defogging system for a neuroendoscopy, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above-mentioned image defogging methods for a neuroendoscopy when executing the computer program.

[0095] In summary, the embodiment of the present application provides an image defogging method for a neuroendoscope, by analyzing the distribution of surgical instruments and human tissues in the operating area in the grayscale image of the neuroendoscope, combining the connected domain and optical flow vector of the grayscale image to construct a moving distance feature value, and combining the color consistency degree in the connected domain and the contour symmetry degree of the connected domain to construct a target suspicion degree. The beneficial effect is that the difference in movement characteristics between surgical instruments and human tissues in the operating area, the color difference between human tissues and surgical instruments in the operating area, and the difference in edge contour symmetry between human tissues and surgical instruments are taken into account, thereby improving the distinction between the corresponding image areas of surgical instruments and human tissues in the operating area in the grayscale image of the neuroendoscope;

[0096] The template block side length is constructed by the target suspicion degree, and the reflective area mask of the neuroendoscopic image is obtained by the threshold segmentation of the HSV image of the neuroendoscopic image; based on the template block side length and the reflective area mask, the reflective area in the neuroendoscopic image is repaired in combination with the Criminisi algorithm, and the image sequence obtained based on the repaired image of the neuroendoscopic image is used to realize the defogging of the repaired neuroendoscopic image using the image defogging method based on the convolutional neural network. The beneficial effect is that the template block size of the pixel points in the reflective area of ​​the neuroendoscopic image is reasonably set in the Criminisi algorithm, while improving the repair effect of the human tissue detail information in the reflective area of ​​the neuroendoscopic image, the repair time of the reflective area caused by the surgical instrument in the neuroendoscopic image is reduced, and the reflective area of ​​the neuroendoscopic image can be effectively repaired in time, so as to more accurately and timely reflect the real situation of the surgical area in the neuroendoscopic image after defogging, thereby improving the defogging effect of the neuroendoscopic image.

[0097] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0099] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for defogging an image of a neuroendoscopy, characterized in that: The method comprises the following steps: Collect each frame of neuroendoscopic image of the surgical area; perform grayscale image and HSV image conversion respectively; Obtain each connected domain in each grayscale image; obtain the optical flow vector of each pixel in each frame of neuroendoscopic image by optical flow estimation algorithm as the optical flow vector of each pixel in the corresponding grayscale image; construct the moving distance feature value of each connected domain based on the overall size and discrete degree of the optical flow vector of all pixels in each connected domain; calculate the color consistency of each connected domain, the expression is: , where Connected domain The color consistency of The standard deviation of the chromaticity values ​​of all pixels corresponding to all pixels in the i-th frame HSV image; Indicates an artificially preset positive number; The gradient direction angle of each edge pixel point in each connected domain is calculated by using a gradient detection algorithm; all the gradient direction angles of each connected domain are statistically histogrammed to obtain each histogram; The degree of symmetry of the outline is recorded as , The calculation expression is: , where p is the connected domain The standard deviation of the height values ​​of all bins in the histogram; a is a manually preset positive number; The fusion value of the moving distance feature value, color consistency and contour symmetry of each connected domain is used as the target suspicion of each connected domain; Determine the target suspicion degree of each pixel in each grayscale image based on the target suspicion degree; perform threshold segmentation on the brightness values ​​of all pixels in each HSV image to obtain each binary image; Calculate the template block side length of the pixel point in the neuroendoscopic image, the template block side length of the u-th pixel point in the i-th frame neuroendoscopic image The calculation expression is: , where f1 and f2 represent the template block size parameters; Indicates the target suspicion of the u-th pixel in the i-th grayscale image; To find the function that is closest to an odd integer; The size of the template block of each pixel is obtained by the side length of the template block of each pixel; Each binary image is used as a mask for the reflective area of ​​each frame of the neuroendoscopic image, and the reflective area in the neuroendoscopic image is repaired using an image repair algorithm in combination with the template block size; an image sequence consisting of repaired images of all frames of the neuroendoscopic image is obtained, and the image sequence is defogged using a neural network.

2. A neuroendoscopic image defogging method as claimed in claim 1, characterized in that: The calculation expression of the moving distance characteristic value of each connected domain is: , where Represents a connected domain The moving distance characteristic value, Represents a connected domain The mean value of the modulus of the optical flow vector of all pixels in ; Represents a connected domain The standard deviation of the optical flow vector modulus of all pixels in Represents the jth connected component of the i-th grayscale image.

3. The method for defogging an image of a neuroendoscopy according to claim 1, characterized in that: The target suspicion degree of each pixel is the target suspicion degree of the connected domain where the pixel is located.

4. The method for defogging an image of a neuroendoscopy according to claim 1, characterized in that: The acquisition process of the repaired image is as follows: Each frame of neuroendoscopic image and the mask of the corresponding reflective area are used as the input of the Criminisi algorithm, wherein the template block size of each pixel point in each frame of neuroendoscopic image is used as the template block size of the pixel point in the Criminisi algorithm, and the repaired image of the neuroendoscopic image is output.

5. The method for defogging an image of a neuroendoscopy according to claim 1, characterized in that: Defogging the image sequence by using a neural network includes: Based on the first convolutional neural network, the original imaging feature map of all neuroendoscopic images in the image sequence is extracted; based on the second convolutional neural network, the smoke feature image sequence of the image sequence is extracted, and based on the smoke feature image sequence, the final smoke feature map of the corresponding neuroendoscopic image is generated; based on the original imaging feature map and the final smoke feature map, the neuroendoscopic image after defogging is obtained.

6. A neuroendoscopic image defogging device, implementing the method as claimed in claim 1, characterized in that: The device comprises: Image acquisition module: collects each frame of neuroendoscopic images of the surgical area; performs grayscale image and HSV image conversion respectively; Image analysis module: obtain each connected domain in each grayscale image; obtain the optical flow vector of each pixel in each frame of neuroendoscopic image through the optical flow estimation algorithm as the optical flow vector of each pixel in the corresponding grayscale image; construct the moving distance feature value of each connected domain based on the overall size and discreteness of the optical flow vectors of all pixels in each connected domain; combine the similarity of the chromaticity values ​​of the pixels in the HSV image corresponding to each connected domain, and the degree of confusion of the gradient direction angle of all edge pixels in each connected domain, to construct the target suspicion of each connected domain; Template block size calculation module: determine the target suspicion of each pixel in each grayscale image based on the target suspicion; perform threshold segmentation on the brightness values ​​of all pixels in each HSV image to obtain each binary image; construct the template block size of each pixel in each frame of neuroendoscopic image based on the target suspicion of each pixel; Image defogging module: each binary image is used as a mask for the reflective area of ​​each frame of the neuroendoscopic image, and the reflective area in the neuroendoscopic image is repaired using an image restoration algorithm in combination with the template block size; an image sequence consisting of the restored images of all frames of the neuroendoscopic image is obtained, and the image sequence is defogged through a neural network.

7. A neuroendoscopic image defogging system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images