Lacrimal river height calculation method, device, equipment, medium and product

Abstract

The invention discloses a lacrimal river height calculation method, device and equipment, a medium and a product, and relates to the field of lacrimal river height, and the method comprises the steps: carrying out the edge detection of an image through an edge detection algorithm, and obtaining an overall edge binary image; the lacrimal river edge in the overall edge binary image is extracted through a manual extraction method, and a complete lacrimal river area binary image is obtained through restoration; combining the manually extracted pupil region binary image and the complete lacrimal river region binary image to obtain a target region binary image; training, verifying and testing the deep convolutional neural network according to each image in the color image sample set and the infrared image sample set and the binary image of the target area to obtain the trained deep convolutional neural network corresponding to the color image and the infrared image; and inputting the to-be-processed image into the trained deep convolutional neural network corresponding to the color image or the infrared image to obtain a target area binary image, and then calculating the height of the tear river, so that the calculation efficiency and precision of the tear river height can be improved.

Description

Technical Field

[0001] The present application relates to the field of tear river height, and in particular to a tear river height calculation method, device, equipment, medium and product. Background Art

[0002] It is well known that when calculating the tear meniscus height, a trained deep convolutional neural network needs to be used to identify the tear meniscus area. However, when training the deep convolutional neural network, a complete manual annotation method is used, and the annotation quality is uneven and the efficiency is low, resulting in limited recognition effect of the deep convolutional neural network trained later, and thus the final calculated tear meniscus height is also limited.

[0003] In summary, a calculation method is needed that can improve the efficiency and accuracy of the calculation of the tear river height. Summary of the invention

[0004] The purpose of this application is to provide a tear river height calculation method, device, equipment, medium and product, which can improve the calculation efficiency and accuracy of the tear river height.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for calculating the height of the tear river, comprising:

[0007] Acquire an image sample set; the image sample set includes a color image sample set and an infrared image sample set; the color image sample set includes a plurality of color images of sample eyes; the infrared image sample set includes a plurality of infrared images of sample eyes;

[0008] For any image in the sample image set, edge detection is performed on the image using an edge detection algorithm to obtain an overall edge binary image of the image;

[0009] Extracting the tear river edge from the overall edge binary map of the image by an artificial extraction method to obtain the tear river edge binary map of the image, and repairing the tear river edge in the tear river edge binary map of the image to obtain a complete tear river region binary map corresponding to the image;

[0010] Extracting the pupil portion of the image by an artificial extraction method to obtain a pupil region binary image of the image;

[0011] Merging the pupil region binary image of the image and the complete tear river region binary image of the image to obtain a target region binary image corresponding to the image;

[0012] The deep convolutional neural network is trained, verified and tested according to each image in the color image sample set and the target area binary image corresponding to each image in the color image sample set to obtain a trained deep convolutional neural network corresponding to the color image;

[0013] The deep convolutional neural network is trained, verified and tested according to each image in the infrared image sample set and the target area binary image corresponding to each image in the infrared image sample set to obtain a trained deep convolutional neural network corresponding to the infrared image;

[0014] If the image to be processed is a color image, the image to be processed is input into a trained deep convolutional neural network corresponding to the color image to obtain a binary image of the target area corresponding to the image to be processed;

[0015] If the image to be processed is an infrared image, the image to be processed is input into a trained deep convolutional neural network corresponding to the infrared image to obtain a binary image of the target area corresponding to the image to be processed;

[0016] The height of the tear river in the image to be processed is calculated according to the target area binary image corresponding to the image to be processed.

[0017] Optionally, performing edge detection on the image by using an edge detection algorithm to obtain an overall edge binary map of the image specifically includes:

[0018] The edge detection of the image is performed using an edge detection algorithm based on the Laplace operator to obtain an overall edge binary map of the image.

[0019] Optionally, repairing the tear river edge in the tear river edge binary image of the image to obtain a complete tear river region binary image corresponding to the image specifically includes:

[0020] The tear river edge in the tear river region binary image of the image is repaired by using a nearest neighbor search algorithm based on KD-Tree to obtain a complete tear river region binary image corresponding to the image.

[0021] Optionally, the deep convolutional neural network includes: a first Unet network, a region of interest extraction layer, and a second Unet network connected in sequence; inputting the image to be processed into a trained deep convolutional neural network to obtain a binary image of the target region corresponding to the image to be processed, specifically including:

[0022] The first Unet network in the trained deep convolutional neural network is used to process the image to be processed to obtain a binary image of the image to be processed;

[0023] Inputting the binary image of the image to be processed and the image to be processed into the region of interest extraction layer in the trained deep convolutional neural network to obtain an image of the region of interest in the image to be processed; the region of interest includes an overall complete pupil region and a longitudinally complete tear meniscus region; the region of interest is a square, the center of the pupil region is located on the vertical symmetry axis of the region of interest, and the vertical symmetry axis passes through the pupil region and the tear meniscus region;

[0024] The second Unet network in the trained deep convolutional neural network is used to process the image of the area of ​​interest in the image to be processed to obtain a binary image of the target area corresponding to the image to be processed.

[0025] Optionally, calculating the height of the tear river in the image to be processed according to the target region binary image corresponding to the image to be processed specifically includes:

[0026] Determine whether the pupil center is obtained by using the Hough circle detection algorithm to identify the pupil center in the target area binary image corresponding to the processed image;

[0027] If the pupil result is not obtained, the centroid of the pupil area in the binary image of the target area corresponding to the image to be processed is determined to be the pupil center in the binary image of the target area corresponding to the image to be processed, and then the height of the tear river in the image to be processed is determined according to the tear river heights corresponding to each pixel point in the left pixel point set and each pixel point in the right pixel point set within the tear river area in the binary image of the target area corresponding to the image to be processed; each pixel point in the left pixel point set is located to the left of the target pixel point, each pixel point in the right pixel point set is located to the right of the target pixel point, and the target pixel point is located in the tear river area in the binary image of the target area corresponding to the image to be processed and has the same horizontal coordinate as the pupil center;

[0028] If the pupil center is obtained, the height of the tear river in the image to be processed is determined according to the tear river heights corresponding to each pixel point in the left pixel point set and each pixel point in the right pixel point set within the tear river area in the target area binary image corresponding to the image to be processed; each pixel point in the left pixel point set is located to the left of the target pixel point, and each pixel point in the right pixel point set is located to the right of the target pixel point, and the target pixel point is located in the tear river area in the target area binary image corresponding to the image to be processed and has the same horizontal coordinate as the pupil center.

[0029] Optionally, edge detection is performed on the image using an edge detection algorithm based on a Laplace operator to obtain an overall edge binary map of the image, specifically:

[0030] Using formula Perform edge detection on the image to obtain an overall edge binary map of the image, where Out represents the overall edge binary map of the image, Input represents the image, and EDOk represents the edge detection operator based on the Laplace operator, represents the convolution operation, a ij represents the data corresponding to the i-th row and j-th column in the edge detection operator based on the Laplace operator, and k represents the size of the edge detection operator based on the Laplace operator.

[0031] In a second aspect, the present application provides a device for calculating the height of a tear river, comprising:

[0032] An acquisition module, used to acquire an image sample set; the image sample set includes a color image sample set and an infrared image sample set; the color image sample set includes a plurality of color images of sample eyes; the infrared image sample set includes a plurality of infrared images of sample eyes;

[0033] An edge detection module is used to perform edge detection on any image in the sample image set by using an edge detection algorithm to obtain an overall edge binary image of the image;

[0034] A complete tear river edge binary map annotation module is used to extract the tear river edge in the overall edge binary map of the image by an artificial extraction method to obtain the tear river edge binary map of the image, and to repair the tear river edge in the tear river edge binary map of the image to obtain a complete tear river area binary map corresponding to the image;

[0035] A pupil area binary map annotation module is used to extract the pupil part in the image by an artificial extraction method to obtain a pupil area binary map of the image;

[0036] A target area binary image annotation module is used to merge the pupil area binary image of the image and the complete tear meniscus area binary image of the image to obtain a target area binary image corresponding to the image;

[0037] A deep convolutional neural network training module trained with color images, used to train, verify and test the deep convolutional neural network according to each image in the color image sample set and the target area binary image corresponding to each image in the color image sample set to obtain a trained deep convolutional neural network corresponding to the color image;

[0038] The infrared image trained deep convolutional neural network training module is used to train, verify and test the deep convolutional neural network according to each image in the infrared image sample set and the target area binary image corresponding to each image in the infrared image sample set to obtain the trained deep convolutional neural network corresponding to the infrared image;

[0039] A processing module, for inputting the image to be processed into a trained deep convolutional neural network corresponding to the color image to obtain a binary image of the target area corresponding to the image to be processed if the image to be processed is a color image; and for inputting the image to be processed into a trained deep convolutional neural network corresponding to the infrared image to obtain a binary image of the target area corresponding to the image to be processed if the image to be processed is an infrared image;

[0040] The tear river height calculation module is used to calculate the height of the tear river in the image to be processed according to the target area binary image corresponding to the image to be processed.

[0041] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the tear river height calculation methods described above.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for calculating the height of the tear river.

[0043] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the tear river height calculation methods described above.

[0044] According to the specific embodiments provided in this application, this application has the following technical effects:

[0045] The present application provides a tear river height calculation method, device, equipment, medium and product. The method performs edge detection on an image through an edge detection algorithm to obtain an overall edge binary map of the image. An artificial extraction method is used to extract the tear river edge in the overall edge binary map of the image and repair it to obtain a complete tear river area binary map corresponding to the image. The manual and machine methods are combined to complete the extraction of the tear river edge, which solves the defect of relying solely on manual labor in the background technology and improves the efficiency and accuracy of tear river height calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a flow chart of a method for calculating the height of the tear river provided in one embodiment of the present application;

[0048] Figure 2 A schematic diagram of a process for obtaining an edge binary map of a sample image set provided by an embodiment of the present application;

[0049] Figure 3 A structural diagram of a deep convolutional neural network provided in one embodiment of the present application;

[0050] Figure 4 A schematic diagram of a tear river height calculation method provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0054] At present, the strategies for improving the accuracy of tear river height calculation in the industry are mostly focused on modifying the network structure. However, the quality of annotation will significantly affect the performance of the trained model. The human-computer interactive annotation method based on edge information extraction can improve the annotation efficiency and has a stable high-quality annotation effect. Therefore, the performance of the algorithm can be significantly improved in terms of annotation quality. Based on this, this application provides a tear river height calculation method, such as Figure 1 and Figure 4 As shown, a method for calculating the height of the tear river is provided, comprising the following steps, wherein:

[0055] Step 201: Obtain an image sample set; the image sample set includes a color image sample set and an infrared image sample set; the color image sample set includes a plurality of sample eye color images; the infrared image sample set includes a plurality of sample eye infrared images. Specifically, obtain color images and infrared images of eye images from multiple medical centers.

[0056] Step 202: For any image in the sample image set, edge detection is performed on the image using an edge detection algorithm to obtain an overall edge binary map of the image.

[0057] Step 203: extract the tear river edge from the overall edge binary map of the image using an artificial extraction method to obtain the tear river edge binary map of the image, and repair the tear river edge in the tear river edge binary map of the image to obtain the complete tear river area binary map corresponding to the image.

[0058] Step 204: extracting the pupil portion from the image using an artificial extraction method to obtain a binary image of the pupil region of the image.

[0059] Step 205: merging the pupil region binary image of the image and the complete tear river region binary image of the image to obtain a target region binary image corresponding to the image.

[0060] Step 206: Train, verify and test the deep convolutional neural network according to each image in the color image sample set and the target area binary map corresponding to each image in the color image sample set to obtain a trained deep convolutional neural network corresponding to the color image. Specifically, each image in the color image sample set and the edge binary map corresponding to each image in the color image sample set are divided into a training set, a verification set and a test set according to 5:3:2.

[0061] Step 207: Train, verify and test the deep convolutional neural network according to each image in the infrared image sample set and the target area binary map corresponding to each image in the infrared image sample set to obtain a trained deep convolutional neural network corresponding to the infrared image. Specifically, each image in the infrared image sample set and the edge binary map corresponding to each image in the infrared image sample set are divided into a training set, a verification set and a test set according to 5:3:2.

[0062] Step 208: If the image to be processed is a color image, the image to be processed is input into the trained deep convolutional neural network corresponding to the color image to obtain a binary image of the target area corresponding to the image to be processed. The image channel information of the image to be processed can be used to automatically identify or manually select whether the image to be processed is an infrared image or a color image.

[0063] Step 209: If the image to be processed is an infrared image, the image to be processed is input into a trained deep convolutional neural network corresponding to the infrared image to obtain a binary image of the target area corresponding to the image to be processed.

[0064] Step 210: Calculate the height of the tear river in the image to be processed according to the target region binary image corresponding to the image to be processed.

[0065] The implementation of the above steps 201 to 210 can improve the calculation efficiency and accuracy of the tear river height.

[0066] In practical applications, before edge detection is performed on the image using an edge detection algorithm to obtain an overall edge binary map of the image, the process further includes:

[0067] The image quality is screened by the trained first deep convolutional neural network (Inception-v3) (used to screen whether the image is clear, whether the through-hole area and tear river area are blocked, etc.). Subsequently, the edge detection algorithm is used to perform edge detection on the image that has passed the quality inspection to obtain the overall edge binary map of the image. The subsequent processing steps are also based on the image that has passed the quality inspection.

[0068] like Figure 2 As shown, in another exemplary embodiment of the present application, edge detection is performed on the image by an edge detection algorithm to obtain an overall edge binary map of the image, which specifically includes:

[0069] The edge detection of the image is performed using an edge detection algorithm based on the Laplace operator to obtain an overall edge binary map of the image.

[0070] In another exemplary embodiment of the present application, repairing the tear river edge in the tear river edge binary image of the image to obtain a complete tear river area binary image corresponding to the image specifically includes:

[0071] The tear river edge in the tear river edge binary image of the image is repaired by using a nearest neighbor search algorithm based on KD-Tree to obtain a complete tear river area binary image corresponding to the image.

[0072] In another exemplary embodiment of the present application, Figure 3 As shown, the deep convolutional neural network includes: a first Unet network, a region of interest extraction layer and a second Unet network connected in sequence.

[0073] Inputting the image to be processed into a trained deep convolutional neural network to obtain a binary image of the target area corresponding to the image to be processed, specifically comprising:

[0074] The first Unet network in the trained deep convolutional neural network is used to process the image to be processed to obtain a binary image of the image to be processed;

[0075] Inputting the binary image of the image to be processed and the image to be processed into the region of interest extraction layer in the trained deep convolutional neural network to obtain an image of the region of interest in the image to be processed; the region of interest includes an overall complete pupil region and a longitudinally complete tear meniscus region; the region of interest is a square, the center of the pupil region is located on the vertical symmetry axis of the region of interest, and the vertical symmetry axis passes through the pupil region and the tear meniscus region;

[0076] The second Unet network in the trained deep convolutional neural network is used to process the image of the area of ​​interest in the image to be processed to obtain a binary image of the target area corresponding to the image to be processed.

[0077] As a more specific implementation, the basic modules required for the deep convolutional neural network include convolution blocks, upsampling layers, downsampling layers, activation layers, and region of interest extraction layers. The convolution block structure is as follows: 3×3 convolution kernel, convolution layer with a step size of 1; batch normalization layer; 0.5 random dropout layer; LeakyRelU activation layer; 3×3 convolution kernel, convolution layer with a step size of 1; batch normalization layer; 0.5 random dropout layer; LeakyRelu activation layer. The upsampling layer structure is as follows: nearest neighbor interpolation layer; 1×1 convolution kernel, convolution layer with a step size of 1; feature fusion layer. The downsampling layer structure is as follows: 3×3 convolution kernel, convolution layer with a step size of 2; batch normalization layer; LeakyRelU activation layer. The activation layer is a Sigmoid activation layer. The number of input channels and output channels of the convolutional layer, upsampling layer, downsampling layer and activation layer are all set manually. The scale of the original data or feature map will be expanded to twice the original size after the upsampling layer, and will be reduced to half of the original size after the downsampling layer.

[0078] The specific architecture of the overall network is as follows:

[0079] The first layer is a convolution block with an input channel number of 1 and an output channel number of 64 as the input layer, and the image to be processed is input.

[0080] The second layer is a downsampling layer with 64 input and output channels.

[0081] The third layer is a convolution block with 64 input channels and 128 output channels as the intermediate feature extraction layer.

[0082] The fourth layer is a downsampling layer with 128 input and output channels.

[0083] The fifth layer is a convolution block with 128 input channels and 256 output channels as the intermediate feature extraction layer.

[0084] The sixth layer is a downsampling layer with 256 input and output channels.

[0085] The seventh layer is a convolution block with 256 input channels and 512 output channels as the intermediate feature extraction layer.

[0086] The eighth layer is a downsampling layer with 512 input and output channels.

[0087] The ninth layer is a convolution block with 512 input channels and 1024 output channels as the intermediate feature extraction layer.

[0088] The tenth layer is an upsampling layer with 1024 input channels and 512 output channels, in which the features of the seventh layer output are concatenated to form an output with 1024 channels.

[0089] The eleventh layer is a convolution block with 1024 input channels and 512 output channels as the intermediate feature extraction layer.

[0090] The twelfth layer is an upsampling layer with 512 input channels and 256 output channels, in which the features of the fifth layer output are concatenated to form an output with 512 channels.

[0091] The thirteenth layer is a convolutional block with 512 input channels and 256 output channels as the intermediate feature extraction layer.

[0092] The fourteenth layer is an upsampling layer with 256 input channels and 128 output channels, in which the features of the output of the third layer are concatenated to form an output with 256 channels.

[0093] The fifteenth layer is a convolutional block with 256 input channels and 128 output channels as the intermediate feature extraction layer.

[0094] The sixteenth layer is an upsampling layer with 128 input channels and 64 output channels, in which the features of the output of the first layer are concatenated and combined into an output with 128 channels.

[0095] The seventeenth layer is a convolutional block with 128 input channels and 64 output channels as the intermediate feature extraction layer.

[0096] The eighteenth layer is a convolutional layer with 64 input channels, 1 output channel, a 3×3 convolution kernel, and a stride of 1.

[0097] The nineteenth layer is the Sigmoid activation layer.

[0098] The 20th layer is the region of interest extraction layer, which is used to input the image to be processed and the feature map output by the 19th layer, and to limit the scale of the image to be processed based on the feature map output by the 19th layer, and use the region of interest as the input of the 21st layer. The function of the region of interest extraction layer is to adjust the scale of the original image based on the output feature map of the previous layer, limit the attention of subsequent modules to the region of interest, and improve the effectiveness of the model parameters.

[0099] The 21st layer is a convolutional block with 1 input channel and 64 output channels as the input layer.

[0100] The 22nd layer is a downsampling layer with 64 input and output channels.

[0101] The 23rd layer is a convolution block with 64 input channels and 128 output channels as the intermediate feature extraction layer.

[0102] The 24th layer is a downsampling layer with 128 input and output channels.

[0103] The 25th layer is a convolution block with 128 input channels and 256 output channels as the intermediate feature extraction layer.

[0104] The 26th layer is a downsampling layer with 256 input and output channels.

[0105] The 27th layer is a convolution block with 256 input channels and 512 output channels as the intermediate feature extraction layer.

[0106] The 28th layer is a downsampling layer with 512 input and output channels.

[0107] The 29th layer is a convolution block with 512 input channels and 1024 output channels as the intermediate feature extraction layer.

[0108] The 30th layer is an upsampling layer with 1024 input channels and 512 output channels, in which the features of the output of the 27th layer are concatenated to form an output with 1024 channels.

[0109] The 31st layer is a convolution block with 1024 input channels and 512 output channels as the intermediate feature extraction layer.

[0110] The 32nd layer is an upsampling layer with 512 input channels and 256 output channels, in which the features of the output of the 25th layer are concatenated to form an output with 512 channels.

[0111] The 33rd layer is a convolution block with 512 input channels and 256 output channels as the intermediate feature extraction layer.

[0112] The 34th layer is an upsampling layer with 256 input channels and 128 output channels, in which the features of the output of the 23rd layer are concatenated to form an output with 256 channels.

[0113] The 35th layer is a convolution block with 256 input channels and 128 output channels as the intermediate feature extraction layer.

[0114] The 36th layer is an upsampling layer with 128 input channels and 64 output channels, in which the features of the output of the 21st layer are concatenated to form an output with 128 channels.

[0115] The 37th layer is a convolution block with 128 input channels and 64 output channels as the intermediate feature extraction layer.

[0116] The 38th layer is a convolutional layer with 64 input channels, 1 output channel, 3×3 convolution kernel, and a stride of 1.

[0117] The 39th layer is the Sigmoid activation layer, which serves as the output layer.

[0118] In another exemplary embodiment of the present application, calculating the height of the tear river in the image to be processed according to the target area binary image corresponding to the image to be processed specifically includes:

[0119] It is determined whether the pupil center is obtained by using the Hough circle detection algorithm to identify the pupil center in the target area binary image corresponding to the image to be processed.

[0120] If the pupil result is not obtained, the centroid of the pupil area in the binary image of the target area corresponding to the image to be processed is determined to be the pupil center in the binary image of the target area corresponding to the image to be processed, and then the height of the tear river in the image to be processed is determined according to the tear river heights corresponding to each pixel in the left pixel set and each pixel in the right pixel set in the tear river area in the binary image of the target area corresponding to the image to be processed; each pixel in the left pixel set is located to the left of the target pixel, and each pixel in the right pixel set is located to the right of the target pixel, and the target pixel is located in the tear river area in the binary image of the target area corresponding to the image to be processed and has the same horizontal coordinate as the pupil center. By setting the parameters in the Hough circle detection algorithm, most of the pupil center coordinates can be located, but when a small number of pupil sizes deviate from the general size, recognition failure may occur. Therefore, the centroid of the pupil area in the edge binary image corresponding to the image to be processed is determined to be the pupil center in the edge binary image corresponding to the image to be processed. Specifically: if the recognition fails, the pupil recognition method based on position information, which is relatively more time-consuming, is used to locate the center coordinates. That is, the pupil recognition method based on position information first detects the highest position of the white area of ​​the binary image of the target area corresponding to the image to be processed, and covers a square area with a side length of 160 pixels from this position downward. The coordinates of the particle in the white area (that is, the center of mass of the pupil area) in the area are identified and regarded as the center coordinates.

[0121] If the pupil center is obtained, the height of the tear river in the image to be processed is determined according to the tear river heights corresponding to each pixel point in the left pixel point set and each pixel point in the right pixel point set within the tear river area in the target area binary image corresponding to the image to be processed; each pixel point in the left pixel point set is located to the left of the target pixel point, and each pixel point in the right pixel point set is located to the right of the target pixel point, and the target pixel point is located in the tear river area in the target area binary image corresponding to the image to be processed and has the same horizontal coordinate as the pupil center.

[0122] In another exemplary embodiment of the present application, determining the height of the tear river in the image to be processed according to the tear river heights corresponding to each pixel point in the left pixel point set and each pixel point in the right pixel point set in the tear river area in the target area binary image corresponding to the image to be processed specifically includes:

[0123] Calculate the average height of the tear river area corresponding to the 5 pixels to the left and right of the target pixel. Calculate, where TMH represents the height of the tear river; ave() represents the average function, y ij Indicates that the horizontal axis is x i In the point set, the vertical coordinates are arranged from large to small, the vertical coordinate of the jth point; the horizontal coordinate of the target pixel point is x k , x k-d Indicates the horizontal coordinate of the dth pixel to the left of the target pixel, x k+d Represents the horizontal coordinate of the d-th pixel to the right of the target pixel, where d is 5.

[0124] In another exemplary embodiment of the present application, edge detection is performed on the image by using an edge detection algorithm based on a Laplacian operator to obtain an overall edge binary map of the image, specifically:

[0125] Using formula Perform edge detection on the image to obtain an overall edge binary map of the image, where Out represents the overall edge binary map of the image, Input represents the image, and EDO k represents the edge detection operator based on the Laplace operator, represents the convolution operation, a ij represents the data corresponding to the i-th row and j-th column in the edge detection operator based on the Laplace operator, and k represents the size of the edge detection operator based on the Laplace operator. Through the edge detection technology based on the Laplace operator, the boundary of the tear river area can be highlighted with the help of image gradient information.

[0126] First, the tear river height data of multiple centers are collected and used, and the data includes two modes: color photos and infrared. During the research process, it was found that there are invisible differences between the data of the two modes. Secondly, the quality and efficiency of mask annotation are improved by edge detection and region repair technology, and this method is called human-computer collaborative annotation method. Then, an attention-restricted convolutional neural network is designed to add a loss function with tightness constraints, and training is performed based on the mask annotation completed by the human-computer collaborative method to achieve synchronous segmentation and fusion of the tear river and pupil center area. Finally, based on the mask obtained by the above neural network segmentation, the pupil center is located and the tear river height is measured in combination with the Hough circle detection. Finally, the development of the method, device and software system for calculating the tear river height is realized.

[0127] In another exemplary embodiment of the present application, after calculating the height of the tear river in the image to be processed according to the target area binary image corresponding to the image to be processed, the method further includes:

[0128] When the tear meniscus height measurement result is greater than 0.64mm or less than 0.02mm, it indicates that the user may have other disease risks or misjudgment, and it is recommended to manually verify the data. The binary image of the target area corresponding to the image to be processed is overlapped and compared with the image to be processed. If the overlap is high, it proves that the calculated tear meniscus height fits the actual value, and the tear meniscus height value and abnormal prompt information are output.

[0129] This application uses a human-computer interactive annotation method, which can improve the annotation efficiency from 3 minutes per image to 40 seconds per image. Using a trained deep convolutional neural network, it only takes 0.2s-0.3s to process an image on average. According to the positioning and measurement algorithm, it only takes 0.2s-0.3s to calculate the height of a tear river on average.

[0130] In the related art, the pupil area and the tear canal area are segmented separately, but since the pupil and tear canal areas are small targets, such a segmentation method can easily reduce the accuracy of the segmentation results. This application improves the processing speed and segmentation quality by fusing the two targets into one target for segmentation. This application can be operated through an interactive interface, which is more user-friendly, and displays the fusion effect diagram, tear canal height measurement results and abnormal prompt information. While focusing on automation, it also focuses on user participation.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a tear river height calculation device for implementing the tear river height calculation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more tear river height calculation device embodiments provided below can refer to the limitations of the tear river height calculation method above, and will not be repeated here.

[0132] In an exemplary embodiment, a tear river height calculation device is provided, comprising:

[0133] The acquisition module is used to acquire an image sample set; the image sample set includes a color image sample set and an infrared image sample set; the color image sample set includes a plurality of color images of sample eyes; the infrared image sample set includes a plurality of infrared images of sample eyes.

[0134] The edge detection module is used to perform edge detection on any image in the sample image set by using an edge detection algorithm to obtain an overall edge binary map of the image.

[0135] The complete tear river edge binary map annotation module is used to extract the tear river edge in the overall edge binary map of the image by using an artificial extraction method to obtain the tear river edge binary map of the image, and repair the tear river edge in the tear river edge binary map of the image to obtain the complete tear river area binary map corresponding to the image.

[0136] The pupil area binary map annotation module is used to extract the pupil part in the image by using an artificial extraction method to obtain a pupil area binary map of the image.

[0137] The target area binary image annotation module is used to merge the pupil area binary image of the image and the complete tear meniscus area binary image of the image to obtain the target area binary image corresponding to the image.

[0138] The deep convolutional neural network training module trained for color images is used to train, verify and test the deep convolutional neural network according to each image in the color image sample set and the target area binary image corresponding to each image in the color image sample set to obtain the trained deep convolutional neural network corresponding to the color image.

[0139] The deep convolutional neural network training module trained for infrared images is used to train, verify and test the deep convolutional neural network according to each image in the infrared image sample set and the binary image of the target area corresponding to each image in the infrared image sample set to obtain the trained deep convolutional neural network corresponding to the infrared image.

[0140] A processing module is used to input the image to be processed into a trained deep convolutional neural network corresponding to the color image to obtain a binary image of the target area corresponding to the image to be processed if the image to be processed is a color image; and to input the image to be processed into a trained deep convolutional neural network corresponding to the infrared image to obtain a binary image of the target area corresponding to the image to be processed if the image to be processed is an infrared image.

[0141] The tear river height calculation module is used to calculate the height of the tear river in the image to be processed according to the target area binary image corresponding to the image to be processed.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for calculating the height of the Tear River. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for calculating the height of the Tear River is implemented.

[0143] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned method embodiments when executing the computer program.

[0144] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0145] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0147] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0148] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0149] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0150] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for calculating the height of the tear river, characterized in that: The tear river height calculation method comprises: Acquire an image sample set; the image sample set includes a color image sample set and an infrared image sample set; the color image sample set includes a plurality of color images of sample eyes; the infrared image sample set includes a plurality of infrared images of sample eyes; For any image in the sample image set, edge detection is performed on the image using an edge detection algorithm to obtain an overall edge binary image of the image; Extracting the tear river edge from the overall edge binary map of the image by an artificial extraction method to obtain the tear river edge binary map of the image, and repairing the tear river edge in the tear river edge binary map of the image to obtain a complete tear river region binary map corresponding to the image; Extracting the pupil portion of the image by an artificial extraction method to obtain a pupil region binary image of the image; Merging the pupil region binary image of the image and the complete tear river region binary image of the image to obtain a target region binary image corresponding to the image; The deep convolutional neural network is trained, verified and tested according to each image in the color image sample set and the target area binary image corresponding to each image in the color image sample set to obtain a trained deep convolutional neural network corresponding to the color image; The deep convolutional neural network is trained, verified and tested according to each image in the infrared image sample set and the target area binary image corresponding to each image in the infrared image sample set to obtain a trained deep convolutional neural network corresponding to the infrared image; If the image to be processed is a color image, the image to be processed is input into a trained deep convolutional neural network corresponding to the color image to obtain a binary image of the target area corresponding to the image to be processed; If the image to be processed is an infrared image, the image to be processed is input into a trained deep convolutional neural network corresponding to the infrared image to obtain a binary image of the target area corresponding to the image to be processed; The height of the tear river in the image to be processed is calculated according to the target area binary image corresponding to the image to be processed.

2. The method for calculating the height of the tear river according to claim 1, characterized in that: Performing edge detection on the image by using an edge detection algorithm to obtain an overall edge binary map of the image specifically includes: The edge detection of the image is performed using an edge detection algorithm based on a Laplace operator to obtain an overall edge binary map of the image.

3. The method for calculating the height of the tear river according to claim 1, characterized in that: Repairing the tear river edge in the tear river edge binary image of the image to obtain a complete tear river region binary image corresponding to the image specifically includes: The tear river edge in the tear river edge binary image of the image is repaired by using a nearest neighbor search algorithm based on KD-Tree to obtain a complete tear river area binary image corresponding to the image.

4. The method for calculating the height of the tear river according to claim 1, characterized in that: The deep convolutional neural network includes: a first Unet network, a region of interest extraction layer, and a second Unet network connected in sequence; the image to be processed is input into the trained deep convolutional neural network to obtain a binary image of the target area corresponding to the image to be processed, specifically including: The first Unet network in the trained deep convolutional neural network is used to process the image to be processed to obtain a binary image of the image to be processed; Inputting the binary image of the image to be processed and the image to be processed into the region of interest extraction layer in the trained deep convolutional neural network to obtain an image of the region of interest in the image to be processed; the region of interest includes an overall complete pupil region and a longitudinally complete tear meniscus region; the region of interest is a square, the center of the pupil region is located on the vertical symmetry axis of the region of interest, and the vertical symmetry axis passes through the pupil region and the tear meniscus region; The second Unet network in the trained deep convolutional neural network is used to process the image of the area of ​​interest in the image to be processed to obtain a binary image of the target area corresponding to the image to be processed.

5. The method for calculating the height of the tear river according to claim 1, characterized in that: Calculating the height of the tear river in the image to be processed according to the target region binary image corresponding to the image to be processed specifically includes: Determine whether the pupil center is obtained by using the Hough circle detection algorithm to identify the pupil center in the target area binary image corresponding to the processed image; If the pupil result is not obtained, the centroid of the pupil area in the binary image of the target area corresponding to the image to be processed is determined to be the pupil center in the binary image of the target area corresponding to the image to be processed, and then the height of the tear river in the image to be processed is determined according to the tear river heights corresponding to each pixel point in the left pixel point set and each pixel point in the right pixel point set within the tear river area in the binary image of the target area corresponding to the image to be processed; each pixel point in the left pixel point set is located to the left of the target pixel point, each pixel point in the right pixel point set is located to the right of the target pixel point, and the target pixel point is located in the tear river area in the binary image of the target area corresponding to the image to be processed and has the same horizontal coordinate as the pupil center; If the pupil center is obtained, the height of the tear river in the image to be processed is determined according to the tear river heights corresponding to each pixel point in the left pixel point set and each pixel point in the right pixel point set within the tear river area in the target area binary image corresponding to the image to be processed; each pixel point in the left pixel point set is located to the left of the target pixel point, and each pixel point in the right pixel point set is located to the right of the target pixel point, and the target pixel point is located in the tear river area in the target area binary image corresponding to the image to be processed and has the same horizontal coordinate as the pupil center.

6. The method for calculating the height of the tear river according to claim 2, characterized in that: The edge detection of the image is performed by an edge detection algorithm based on the Laplace operator to obtain an overall edge binary map of the image, which is specifically: Using formula Perform edge detection on the image to obtain an overall edge binary map of the image, where Out represents the overall edge binary map of the image, Input represents the image, and EDO k represents the edge detection operator based on the Laplace operator, represents the convolution operation, a ij represents the parameter corresponding to the i-th row and j-th column in the edge detection operator based on the Laplace operator, and k represents the size of the edge detection operator based on the Laplace operator.

7. A device for calculating the height of the tear river, characterized in that: The tear river height calculation device comprises: An acquisition module, used to acquire an image sample set; the image sample set includes a color image sample set and an infrared image sample set; the color image sample set includes a plurality of color images of sample eyes; the infrared image sample set includes a plurality of infrared images of sample eyes; An edge detection module is used to perform edge detection on any image in the sample image set by using an edge detection algorithm to obtain an overall edge binary image of the image; A complete tear river region binary map annotation module is used to extract the tear river edge in the overall edge binary map of the image by an artificial extraction method to obtain the tear river edge binary map of the image, and to repair the tear river edge in the tear river edge binary map of the image to obtain the complete tear river region binary map corresponding to the image; A pupil area binary map annotation module is used to extract the pupil part in the image by an artificial extraction method to obtain a pupil area binary map of the image; A target area binary image annotation module is used to merge the pupil area binary image of the image and the complete tear meniscus area binary image of the image to obtain a target area binary image corresponding to the image; A deep convolutional neural network training module trained with color images, used to train, verify and test the deep convolutional neural network according to each image in the color image sample set and the target area binary image corresponding to each image in the color image sample set to obtain a trained deep convolutional neural network corresponding to the color image; The deep convolutional neural network training module trained with infrared images is used to train, verify and test the deep convolutional neural network according to each image in the infrared image sample set and the target area binary image corresponding to each image in the infrared image sample set to obtain the trained deep convolutional neural network corresponding to the infrared image; A processing module, for inputting the image to be processed into a trained deep convolutional neural network corresponding to the color image to obtain a binary image of the target area corresponding to the image to be processed if the image to be processed is a color image; and for inputting the image to be processed into a trained deep convolutional neural network corresponding to the infrared image to obtain a binary image of the target area corresponding to the image to be processed if the image to be processed is an infrared image; The tear river height calculation module is used to calculate the height of the tear river in the image to be processed according to the target area binary image corresponding to the image to be processed.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the tear river height calculation method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the tear river height calculation method described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the tear river height calculation method described in any one of claims 1 to 6 is implemented.

Images