Method, device and system for extracting eyelid fissure based on texture-aware neural network

Abstract

The present invention relates to a method, device and system for extracting eyelid fissures based on texture-aware neural networks. The method comprises: obtaining an eye movement image of a user; inputting the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image, wherein the pre-trained segmentation model is a TAU-Net texture-aware neural network, which uses a U-Net network as a backbone network and adds three different subtractive convolution modules. The present invention accurately detects the boundary of the eyelid fissure and has important clinical application value.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and more specifically, to a method, device, system and computer-readable storage medium for extracting eyelid fissures based on a texture-aware neural network. Background Art

[0002] To identify incomplete blinks, the commonly used strategy is to use high-speed video or medical equipment to capture eye movements over a period of time, and the videos are manually processed frame by frame by clinicians. Ophthalmologists can help determine the start and end time of an incomplete blink, as well as its pattern or attributes, however, it is time-consuming and prone to miss some incomplete blinks due to the short blink cycle. In addition, the manual process usually involves relatively high inter- and intra-reader variability because there are no clear standards or quantitative indicators to distinguish incomplete blinks from complete blinks. In order to accurately and effectively identify incomplete blinks, it is desirable to develop a computer tool to automatically extract the palpebral fissure from each frame of eye video, as blinks are closely related to the width of the palpebral fissure. Compared with other types of medical images, eye videos have some significant features that may pose challenges to segmentation, including the complex and irrelevant background around the palpebral fissure, which may cause confusion between the background and the fissure, making it challenging to accurately detect the boundaries of the palpebral fissure, and few available neural networks have special considerations for these features, which may lead to unsatisfactory segmentation performance. Summary of the invention

[0003] To solve the above problems, we establish a method, device, system and computer-readable storage medium for extracting eyelid fissure based on texture-aware neural network.

[0004] The present application discloses a method for extracting eyelid fissure based on texture-aware neural network, the method comprising:

[0005] Acquire an eye movement image of the user;

[0006] Inputting the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image;

[0007] The pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder including a first encoding module and five groups of second encoding modules, wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules are connected to an upsampling block, the first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the second encoding module is composed of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, and a merging layer; the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to an upsampling block, and optionally, each group of decoding modules is composed of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, and a merging layer;

[0008] Optionally, the second encoding module consists of a third convolution layer, a fourth convolution layer, a short subtraction convolution block, a medium subtraction convolution block, and a merging layer;

[0009] Optionally, each group of decoding modules consists of a fifth convolution layer, a sixth convolution layer, a short subtraction convolution block, a medium subtraction convolution block, and a merging layer;

[0010] Optionally, each group of decoding modules consists of a fifth convolution layer, a sixth convolution layer, a short subtraction convolution block, a medium subtraction convolution block, a long subtraction convolution block, and a merging layer;

[0011] The eye movement image is input into an encoder of a pre-trained segmentation model to extract image features, and the decoder is used to receive the image features extracted by the encoder, so as to generate a target area included between the upper and lower eyelids in the eye movement image.

[0012] Further, the first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer. The feature map is input into the first encoding module, a first convolutional feature is obtained through the first convolutional layer, the first convolutional feature is input into the second convolutional layer to obtain a second convolutional feature, the first convolutional feature and the second convolutional feature are input into the short subtraction convolution block to obtain a short subtraction convolution feature, the short subtraction convolution feature and the second convolution feature are input into the merging layer to obtain a first encoding merged feature;

[0013] Optionally, the first encoding merge feature is input into a downsampling block to obtain a first encoding module feature.

[0014] Further, the second encoding module is composed of a third convolution layer, a fourth convolution layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer. The feature of the previous encoding module is input into the second encoding module, and the third convolution feature is obtained through the third convolution layer. The third convolution feature is input into the fourth convolution layer to obtain a fourth convolution feature. The third convolution feature and the fourth convolution feature are input into the short subtraction convolution block to obtain a short subtraction convolution feature. The fourth convolution feature and the second convolution feature or the fourth convolution feature of the previous encoding module are input into the middle subtraction convolution block to obtain a middle subtraction convolution feature. The fourth convolution feature, the short subtraction convolution feature and the middle subtraction convolution feature are input into the merging layer to obtain a second encoding merged feature.

[0015] Optionally, the second merged feature is input into a downsampling block to obtain a second encoding module feature.

[0016] Further, the first encoding module features are input into the first group of second encoding modules to obtain a first group of second encoding merged features, and the first group of second encoding merged features are input into the downsampling block to obtain a first group of second encoding module features;

[0017] Optionally, the first group of second encoding module features are input into the next group of second encoding modules to obtain third encoding merged features, and the third merged features are input into the downsampling block to obtain third encoding module features;

[0018] Optionally, the third encoding module feature is input into the next group of second encoding modules to obtain a fourth encoding merge feature, and the fourth merge feature is input into the downsampling block to obtain a fourth encoding module feature;

[0019] Optionally, the fourth encoding module feature is input into the next group of second encoding modules to obtain a fifth encoding merge feature, and the fifth merge feature is input into the downsampling block to obtain a fifth encoding module feature;

[0020] Optionally, the fifth encoding module feature is input into the next group of second encoding modules to obtain a sixth encoding merge feature, and the sixth merge feature is input into the upsampling block to obtain a sixth encoding module feature.

[0021] Further, the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to the upsampling block, each group of decoding modules consists of a fifth convolution layer, a sixth convolution layer, a short subtraction convolution block, a medium subtraction convolution block, a long subtraction convolution block, and a merging layer, the sixth encoding module feature is input into the first group of decoding modules, the fifth convolution feature is obtained through the fifth convolution layer, the fifth convolution feature is input into the sixth convolution layer to obtain the sixth convolution feature, the fifth convolution feature and the sixth convolution feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature, the fourth convolution feature and the sixth convolution feature of the fourth group of second encoding modules are input into the medium subtraction convolution block to obtain the medium subtraction convolution feature, the fourth convolution feature and the sixth convolution feature of the fourth group of second encoding modules are input into the long subtraction convolution block to obtain the long subtraction convolution feature, the sixth convolution feature, the short subtraction convolution feature, the medium subtraction convolution feature, and the long subtraction convolution feature are input into the merging layer to obtain the first decoding merged feature;

[0022] Optionally, the first decoding merges the features and inputs them into an upsampling block to obtain data reconstructed by the first decoding module;

[0023] Optionally, the data reconstructed by the first decoding module and the fourth convolutional feature of the encoding module at the same level are input into the second group of decoding modules to obtain second decoding merged features, and the second decoding merged features are input into the upsampling block to obtain data reconstructed by the second decoding module;

[0024] Optionally, the data reconstructed by the second decoding module and the fourth convolutional feature of the encoding module at the same level are input into a third group of decoding modules to obtain a third decoding merged feature, and the third decoding merged feature is input into an upsampling block to obtain data reconstructed by the third decoding module;

[0025] Optionally, the data reconstructed by the third decoding module and the fourth convolutional features of the encoding module at the same level are input into a fourth group of decoding modules to obtain a fourth decoding merged feature, and the fourth decoding merged feature is input into an upsampling block to obtain data reconstructed by the fourth decoding module;

[0026] Optionally, the data reconstructed by the fourth decoding module and the second convolution feature of the encoding module at the same level are input into the fifth group of decoding modules to obtain the fifth decoding merged feature, and the fifth decoding merged feature undergoes a convolution, batch normalization and activation function activation operation to generate the target area between the upper and lower eyelids in the eye motion image.

[0027] Further, the short subtractive convolution block consists of two element subtraction and absolute value operation. Optionally, the short subtractive convolution block consists of two element subtraction, absolute value operation and sigmoid activation; Optionally, the short subtractive convolution block consists of element subtraction, absolute value operation, element addition operation and sigmoid activation; Optionally, the medium subtractive convolution block consists of a sampling operation, two 1×1 spatial convolutions with BN and ReLU, and a short subtractive convolution block; Optionally, the long subtractive convolution block consists of two element subtraction, absolute value operation and sigmoid activation;

[0028] Optionally, the merging layer is a concate module;

[0029] Optionally, the downsampling block is a 2× 2 MaxPooling layer;

[0030] Optionally, the upsampling block is a 2× 2 upsampling layer.

[0031] Furthermore, the batch normalization may be selected from one or more of the following: Sync BatchNorm, LayerNormalization, BatchNorm; optionally, the activation function may be selected from one or more of the following: sigmoid, tanh, and Relu.

[0032] A device for extracting eyelid fissure based on texture perception neural network, the device comprising: a memory and a processor;

[0033] The memory is used to store program instructions;

[0034] The processor is used to call program instructions, and when the program instructions are executed, it is used to execute the above-mentioned method for extracting eyelid fissure based on texture perception neural network.

[0035] A device for extracting eyelid fissure based on texture perception neural network, comprising:

[0036] An acquisition unit, used for acquiring an eye movement image of a user;

[0037] A processing unit, configured to input the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image;

[0038] The pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder including a first encoding module and five groups of second encoding modules, wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules are connected to an upsampling block, the first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the second encoding module is composed of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer; the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to an upsampling block, and each group of decoding modules is composed of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, a middle subtraction convolution block, a long subtraction convolution block, and a merging layer;

[0039] The eye movement image is input into an encoder of a pre-trained segmentation model to extract image features, and the decoder is used to receive the image features extracted by the encoder, so as to generate a target area included between the upper and lower eyelids in the eye movement image.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for extracting eyelid fissures based on a texture-aware neural network.

[0041] Advantages of this application:

[0042] 1. This application innovatively discloses a new U-NET neural network (TAU-Net), which uses the U-Net network as the backbone network and adds three different subtractive convolution modules SCB (short subtraction block, medium subtraction block, long subtraction block) for the encoding and decoding process in the U-Net network, aiming to detect various fine structural patterns related to the target object, making the new U-NET neural network more sensitive to the boundary of the detected object;

[0043] 2. This application creatively applies a new U-NET neural network to extract the palpebral fissure and accurately detect the boundary of the palpebral fissure, overcoming the problem of improper segmentation caused by confusion between the background and the fissure due to the complex and irrelevant background around the palpebral fissure when using the existing segmentation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a schematic flow chart of a method for extracting eyelid fissure based on a texture-aware neural network provided by an embodiment of the present invention;

[0046] Figure 2 Schematic diagram of a device for extracting eyelid fissure based on a texture-aware neural network provided by an embodiment of the present invention;

[0047] Figure 3 is a schematic flow chart of a device for extracting eyelid fissure based on a texture-aware neural network provided by an embodiment of the present invention;

[0048] Figure 4 It is a composition and connection diagram of a first encoding module and a first group of second encoding modules provided in an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of the composition and connection of a decoding module provided in an embodiment of the present invention;

[0050] Figure 6 is a schematic diagram of the architecture of TAU-Net provided by an embodiment of the present invention;

[0051] Figure 7 Schematic diagram of the composition and connection of the short subtractive convolution block and the long subtractive convolution block provided in the embodiment of the present invention

[0052] Figure 8 It is a composition and connection diagram of the subtractive convolution block provided in an embodiment of the present invention;

[0053] Fig. 9 is a performance comparison result diagram between the networks provided by the embodiment of the present invention, the first two rows are given frame images and their manual annotations, and the last seven rows are the segmentation results obtained by U-Net, ATNet, AS-Net, Bio-Net, BG-CNN, DeepLabv3 and TAU-Net;

[0054] Fig.10 9 are the segmentation results of TAU-Net and its five variants provided by an embodiment of the present invention; from top to bottom, they are the results of TAU-Net, TAU-Net1, TAU-Net2, TAU-Net3, TAU-Net4 and TAU-Net5 for the given frame image shown in FIG9 . DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0056] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] Figure 1 1 is a schematic flow chart of a method for extracting eyelid fissures based on a texture-aware neural network provided by an embodiment of the present invention. Specifically, the method comprises the following steps:

[0059] 101: Acquire an eye movement image of a user;

[0060] In one embodiment, to identify incomplete blinks, we collected 60 2-minute videos from the eyes of 60 participants (30 dry eye patients and 30 healthy subjects) using a Keratograph 5M (Oculus, Germany) from Wenzhou Medical University Eye Hospital. These videos were acquired in infrared light mode at 8 frames per second (FPS). Each video contains approximately 1,000 frames of images with the same size of 1,360× 1,024 pixels. 1,196 frames of images were randomly selected from these videos as eye movement images for model training.

[0061] 102: Input the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image; the pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder includes a first encoding module and five groups of second encoding modules, wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules is connected to an upsampling block, the first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the The second encoding module consists of a third convolution layer, a fourth convolution layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer; the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to an upsampling block, and each group of decoding modules consists of a fifth convolution layer, a sixth convolution layer, a short subtraction convolution block, a middle subtraction convolution block, a long subtraction convolution block, and a merging layer; the eye movement image is input into an encoder of a pre-trained segmentation model to extract image features, and the decoder is used to receive the image features extracted by the encoder, thereby generating a target area contained between the upper and lower eyelids in the eye movement image.

[0062] In one embodiment, the first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer. The feature map is input into the first encoding module, and the first convolutional feature is obtained through the first convolutional layer. The first convolutional feature is input into the second convolutional layer to obtain the second convolutional feature. The first convolutional feature and the second convolutional feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature. The short subtraction convolution feature and the second convolution feature are input into the merging layer to obtain the first encoding merged feature; optionally, the first encoding merged feature is input into the downsampling block to obtain the first encoding module feature.

[0063] In one embodiment, the second encoding module is composed of a third convolution layer, a fourth convolution layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer. The feature of the previous encoding module is input into the second encoding module, and the third convolution feature is obtained through the third convolution layer. The third convolution feature is input into the fourth convolution layer to obtain the fourth convolution feature. The third convolution feature and the fourth convolution feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature. The fourth convolution feature and the second convolution feature or the fourth convolution feature of the previous encoding module are input into the middle subtraction convolution block to obtain the middle subtraction convolution feature. The fourth convolution feature, the short subtraction convolution feature and the middle subtraction convolution feature are input into the merging layer to obtain the second encoding merged feature. Optionally, the second merged feature is input into the downsampling block to obtain the second encoding module feature.

[0064] In one embodiment, Figure 4The composition and connection of the first encoding module and the first group of second encoding modules are shown. The first encoding module consists of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer. The first encoding module is connected to a downsampling block. The second encoding module consists of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, a medium subtraction convolution block, and a merging layer. The second encoding module is connected to a downsampling block.

[0065] In one embodiment, the first encoding module feature is input into the first group of second encoding modules to obtain the first group of second encoding merged features, and the first group of second encoding merged features is input into the downsampling block to obtain the first group of second encoding module features; optionally, the first group of second encoding module features is input into the next group of second encoding modules to obtain the third encoding merged features, and the third merged features are input into the downsampling block to obtain the third encoding module features; optionally, the third encoding module features are input into the next group of second encoding modules to obtain the fourth encoding merged features, and the fourth merged features are input into the downsampling block to obtain the fourth encoding module features; optionally, the fourth encoding module features are input into the next group of second encoding modules to obtain the fifth encoding merged features, and the fifth merged features are input into the downsampling block to obtain the fifth encoding module features; optionally, the fifth encoding module features are input into the next group of second encoding modules to obtain the sixth encoding merged features, and the sixth merged features are input into the upsampling block to obtain the sixth encoding module features.

[0066] In one embodiment, the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to the upsampling block, each group of decoding modules consists of a fifth convolution layer, a sixth convolution layer, a short subtraction convolution block, a medium subtraction convolution block, a long subtraction convolution block, and a merging layer. The sixth encoding module feature is input into the first group of decoding modules, and the fifth convolution feature is obtained through the fifth convolution layer. The fifth convolution feature is input into the sixth convolution layer to obtain the sixth convolution feature. The fifth convolution feature and the sixth convolution feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature. The fourth convolution feature and the sixth convolution feature of the fourth group of second encoding modules are input into the medium subtraction convolution block to obtain the medium subtraction convolution feature. The fourth convolution feature and the sixth convolution feature of the fourth group of second encoding modules are input into the long subtraction convolution block to obtain the long subtraction convolution feature. The sixth convolution feature, the short subtraction convolution feature, the medium subtraction convolution feature, and the long subtraction convolution feature are input into the merging layer to obtain the first decoding merged feature; optionally, the first decoding merged feature is input into the upsampling block to obtain the data reconstructed by the first decoding module.

[0067] In one embodiment, Figure 5 The composition and connection of the decoding module shown in the figure, the decoding module consists of the fifth convolution layer, the sixth convolution layer, the short subtraction convolution block, the medium subtraction convolution block, the long subtraction convolution block, and the merging layer. The first four groups of decoding modules are connected to the upsampling block.

[0068] In one embodiment, the data reconstructed by the first decoding module and the fourth convolution feature of the encoding module at the same level are input into the second group of decoding modules to obtain the second decoding merge feature, and the second decoding merge feature is input into the upsampling block to obtain the data reconstructed by the second decoding module; optionally, the data reconstructed by the second decoding module and the fourth convolution feature of the encoding module at the same level are input into the third group of decoding modules to obtain the third decoding merge feature, and the third decoding merge feature is input into the upsampling block to obtain the data reconstructed by the third decoding module; optionally, the data reconstructed by the third decoding module and the fourth convolution feature of the encoding module at the same level are input into the fourth group of decoding modules to obtain the fourth decoding merge feature, and the fourth decoding merge feature is input into the upsampling block to obtain the data reconstructed by the fourth decoding module; optionally, the data reconstructed by the fourth decoding module and the second convolution feature of the encoding module at the same level are input into the fifth group of decoding modules to obtain the fifth decoding merge feature, and the fifth decoding merge feature is subjected to a convolution, batch normalization and activation function activation operation to generate the target area contained between the upper and lower eyelids in the eye motion image.

[0069] In one embodiment, the short subtractive convolution block (e.g. Figure 7 ) consists of two element-wise subtraction, absolute value operation and sigmoid activation; Optionally, the subtractive convolution block (such as Figure 8 ) consists of a sampling operation, two 1× 1 spatial convolutions with BN and ReLU, and a short subtractive convolution block; optionally, the long subtractive convolution block (as shown in Figure 7 As shown in the figure, the method comprises the subtraction of two elements, the absolute value operation and the sigmoid activation; optionally, the merging layer is a concate module; optionally, the downsampling block is a 2× 2 MaxPooling layer; optionally, the upsampling block is a 2× 2 upsampling layer.

[0070] In one embodiment, the batch normalization can be selected from one or more of the following: Sync BatchNorm, Layer Normalization, BatchNorm; optionally, the activation function can be selected from one or more of the following: sigmoid, tanh and Relu.

[0071] In one embodiment, Figure 6Schematic diagram of the architecture of the TAU-Net, the pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder includes a first encoding module, five groups of second encoding modules (the first group of second encoding modules, the second group of second encoding modules, the third group of second encoding modules, the fourth group of second encoding modules, and the fifth group of second encoding modules), wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules are connected to an upsampling block, the first encoding module consists of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the second encoding module consists of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, a medium subtraction convolution block, and a merging layer; the decoder includes five groups of decoding modules (the first group of decoding modules, the second group of decoding modules, the third group of decoding modules, the fourth group of decoding modules, and the fifth group of decoding modules), wherein the first four groups of decoding modules are connected to an upsampling block, and each group of decoding modules consists of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, a medium subtraction convolution block, a long subtraction convolution block, and a merging layer. The same-level encoding modules refer to the same-level encoding modules of the first group of decoding modules are the fourth group of second encoding modules, the same-level encoding modules of the second group of decoding modules are the third group of second encoding modules, the same-level encoding modules of the third group of decoding modules are the second group of second encoding modules, the same-level encoding modules of the fourth group of decoding modules are the first group of second encoding modules, and the same-level encoding modules of the fifth group of decoding modules are the first encoding modules.

[0072] In one embodiment, we comprehensively evaluate the performance of the developed TAU-Net for extracting palpebral fissures from frame images using four different metrics, including Dice, intersection over union (IOU), structural similarity (SSIM), and Hausdorff distance (in pixels). Based on these metrics, we perform TAU-Net six times by randomly permuting the three parts of the collected dataset for training, validation, and testing, respectively, and then compare it with several existing networks (i.e., U-Net, AttentionU-Net (AT-Net), Asymmetric U-Net (AS-Net), BiO-Net, BG-CNN, and DeepLabv3 with xception backbone) using the same experimental configuration, and the results are shown in Figure 2. Fig. 9As shown. In addition, we evaluate the effect of the introduced SCB (Subtractive Convolution Block) by removing different components from the developed network. Therefore, we have five different variants, called TAU-Net1, TAU-Net2, TAU-Net3, TAU-Net4 and TAU-Net5. TAU-Net1, TAU-Net2 and TAU-Net3 are defined by removing the long subtractive convolution block, the medium subtractive convolution block, and the medium and long subtractive convolution blocks from the developed network, respectively; TAU-Net4 is defined by removing the sigmoid activation from the short subtractive convolution block, and TAU-Net5 is constructed by replacing the second subtraction operation in the short subtractive convolution block with an element-wise addition operation. The results are shown in Fig.10 shown.

[0073] Figure 2 A device for extracting eyelid fissure based on a texture-aware neural network provided by an embodiment of the present invention includes: a memory and a processor;

[0074] The memory is used to store program instructions;

[0075] The processor is used to call program instructions, and when the program instructions are executed, it is used to execute the above-mentioned method for extracting eyelid fissure based on texture perception neural network.

[0076] Figure 3 A device for extracting eyelid fissure based on texture perception neural network, comprising:

[0077] An acquisition unit, used for acquiring an eye movement image of a user;

[0078] A processing unit, configured to input the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image;

[0079] The pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder including a first encoding module and five groups of second encoding modules, wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules are connected to an upsampling block, the first encoding module consists of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the second encoding module consists of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer; the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to an upsampling block, and each group of decoding modules consists of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, a middle subtraction convolution block, a long subtraction convolution block, and a merging layer; the eye movement image is input into the encoder of the pre-trained segmentation model to extract image features, and the decoder is used to receive the image features extracted by the encoder, thereby generating a target area contained between the upper and lower eyelids in the eye movement image.

[0080] An object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for extracting palpebral fissure based on texture-aware neural network.

[0081] The validation results of this validation example show that assigning intrinsic weights to indications can moderately improve the performance of the method relative to the default setting.

[0082] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0083] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0087] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0088] The above is a detailed introduction to a computer device provided by the present invention. For a person skilled in the art, according to the concept of the embodiments of the present invention, there may be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for extracting eyelid fissure based on texture-aware neural network, characterized in that: The method comprises: Acquire an eye movement image of the user; Inputting the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image; The pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder includes a first encoding module and five groups of second encoding modules, wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules is connected to an upsampling block, the first encoding module consists of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the second encoding module consists of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, and a merging layer; the decoder includes five groups A decoding module, wherein the first four groups of decoding modules are connected to the upsampling block, and each group of decoding modules consists of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, and a merging layer; the second encoding module consists of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, a medium subtraction convolution block, and a merging layer; each group of decoding modules consists of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, a medium subtraction convolution block, and a merging layer; each group of decoding modules consists of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, a medium subtraction convolution block, a long subtraction convolution block, and a merging layer.

2. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer. The feature map is input into the first encoding module, and the first convolutional feature is obtained through the first convolutional layer. The first convolutional feature is input into the second convolutional layer to obtain the second convolutional feature. The first convolutional feature and the second convolutional feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature. The short subtraction convolution feature and the second convolution feature are input into the merging layer to obtain the first encoded merged feature.

3. The method for extracting eyelid fissure based on texture-aware neural network according to claim 2, characterized in that: The first encoding merge feature is input into the downsampling block to obtain the first encoding module feature.

4. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The second encoding module is composed of a third convolution layer, a fourth convolution layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer. The feature of the previous encoding module is input into the second encoding module, and the third convolution feature is obtained through the third convolution layer. The third convolution feature is input into the fourth convolution layer to obtain the fourth convolution feature. The third convolution feature and the fourth convolution feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature. The fourth convolution feature and the second convolution feature or the fourth convolution feature of the previous encoding module are input into the middle subtraction convolution block to obtain the middle subtraction convolution feature. The fourth convolution feature, the short subtraction convolution feature and the middle subtraction convolution feature are input into the merging layer to obtain the second encoding merged feature.

5. The method for extracting eyelid fissure based on texture-aware neural network according to claim 4, characterized in that: The second encoding merge feature is input into the downsampling block to obtain the second encoding module feature.

6. The method for extracting eyelid fissure based on texture-aware neural network according to claim 3, characterized in that: The first encoding module features are input into the first group of second encoding modules to obtain a first group of second encoding merged features, and the first group of second encoding merged features are input into the downsampling block to obtain a first group of second encoding module features.

7. The method for extracting eyelid fissure based on texture-aware neural network according to claim 6, characterized in that: The first group of second encoding module features are input into the next group of second encoding modules to obtain third encoding merge features, and the third encoding merge features are input into the downsampling block to obtain third encoding module features.

8. The method for extracting eyelid fissure based on texture-aware neural network according to claim 7, characterized in that: The third encoding module feature is input into the next group of second encoding modules to obtain a fourth encoding merge feature, and the fourth encoding merge feature is input into the downsampling block to obtain a fourth encoding module feature.

9. The method for extracting eyelid fissure based on texture-aware neural network according to claim 8, characterized in that: The fourth encoding module feature is input into the next group of second encoding modules to obtain the fifth encoding merge feature, and the fifth encoding merge feature is input into the downsampling block to obtain the fifth encoding module feature.

10. The method for extracting eyelid fissure based on texture-aware neural network according to claim 9, characterized in that: The fifth encoding module feature is input into the next group of second encoding modules to obtain the sixth encoding merge feature, and the sixth encoding merge feature is input into the upsampling block to obtain the sixth encoding module feature.

11. The method for extracting eyelid fissure based on texture-aware neural network according to claim 10, characterized in that: The decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to the upsampling block, each group of decoding modules consists of a fifth convolution layer, a sixth convolution layer, a short subtraction convolution block, a medium subtraction convolution block, a long subtraction convolution block, and a merging layer; the sixth encoding module feature is input into the first group of decoding modules, and the fifth convolution feature is obtained through the fifth convolution layer; the fifth convolution feature is input into the sixth convolution layer to obtain the sixth convolution feature; the fifth convolution feature and the sixth convolution feature are input into the short subtraction convolution block to obtain the short subtraction convolution feature; the fourth convolution feature and the sixth convolution feature of the fourth group of second encoding modules are input into the medium subtraction convolution block to obtain the medium subtraction convolution feature; the fourth convolution feature and the sixth convolution feature of the fourth group of second encoding modules are input into the long subtraction convolution block to obtain the long subtraction convolution feature; the sixth convolution feature, the short subtraction convolution feature, the medium subtraction convolution feature, and the long subtraction convolution feature are input into the merging layer to obtain the first decoding merged feature.

12. The method for extracting eyelid fissure based on texture-aware neural network according to claim 11, characterized in that: The first decoding merged features are input into the upsampling block to obtain data reconstructed by the first decoding module.

13. The method for extracting eyelid fissure based on texture-aware neural network according to claim 12, characterized in that: The data reconstructed by the first decoding module and the fourth convolutional feature of the encoding module at the same level are input into the second group of decoding modules to obtain the second decoding merged feature, and the second decoding merged feature is input into the upsampling block to obtain the data reconstructed by the second decoding module.

14. The method for extracting eyelid fissure based on texture-aware neural network according to claim 13, characterized in that: The data reconstructed by the second decoding module and the fourth convolutional feature of the encoding module at the same level are input into the third group of decoding modules to obtain the third decoding merged feature, and the third decoding merged feature is input into the upsampling block to obtain the data reconstructed by the third decoding module.

15. The method for extracting eyelid fissure based on texture-aware neural network according to claim 14, characterized in that: The data reconstructed by the third decoding module and the fourth convolutional features of the encoding module at the same level are input into the fourth group of decoding modules to obtain the fourth decoding merged features, and the fourth decoding merged features are input into the upsampling block to obtain the data reconstructed by the fourth decoding module.

16. The method for extracting eyelid fissure based on texture-aware neural network according to claim 15, characterized in that: The data reconstructed by the fourth decoding module and the second convolution feature of the encoding module at the same level are input into the fifth group of decoding modules to obtain the fifth decoding merged feature. The fifth decoding merged feature undergoes a convolution, batch normalization and activation function activation operation to generate the target area between the upper and lower eyelids in the eye motion image.

17. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The short subtraction convolution block consists of two element subtraction and absolute value operation, or the short subtraction convolution block consists of two element subtraction, absolute value operation and sigmoid activation; or the short subtraction convolution block consists of element subtraction, absolute value operation, element addition operation and sigmoid activation.

18. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The subtractive convolution block consists of a sampling operation, two 1×1 spatial convolutions with BN and ReLU, and a short subtractive convolution block.

19. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The long subtraction convolution block consists of two element-wise subtraction, absolute value operation and sigmoid activation.

20. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The merging layer is a concate module.

21. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The downsampling block is a 2×2 MaxPooling layer.

22. The method for extracting eyelid fissure based on texture-aware neural network according to claim 1, characterized in that: The upsampling block is a 2× 2 upsampling layer.

23. The method for extracting eyelid fissure based on texture-aware neural network according to claim 16, characterized in that: The batch normalization is selected from one or more of the following: Sync BatchNorm, Layer Normalization, BatchNorm.

24. The method for extracting eyelid fissure based on texture-aware neural network according to claim 16, characterized in that: The activation function is selected from one or more of the following: sigmoid, tanh and Relu.

25. A device for extracting eyelid fissure based on texture-aware neural network, the device comprising: Memory and processor; The memory is used to store program instructions; The processor is used to call program instructions, and when the program instructions are executed, it is used to execute the method for extracting palpebral fissure based on texture perception neural network as described in any one of claims 1-24.

26. A device for extracting eyelid fissure based on texture perception neural network, comprising: An acquisition unit, used for acquiring an eye movement image of a user; A processing unit, configured to input the eye movement image into a pre-trained segmentation model to obtain a target area between the upper and lower eyelids in the eye movement image; The pre-trained segmentation model is a TAU-Net texture perception neural network, including an encoder and a decoder, the encoder including a first encoding module and five groups of second encoding modules, wherein the first encoding module and the first four groups of second encoding modules are connected to a downsampling block, and the fifth group of second encoding modules are connected to an upsampling block, the first encoding module is composed of a first convolutional layer, a second convolutional layer, a short subtraction convolution block, and a merging layer, and the second encoding module is composed of a third convolutional layer, a fourth convolutional layer, a short subtraction convolution block, a middle subtraction convolution block, and a merging layer; the decoder includes five groups of decoding modules, wherein the first four groups of decoding modules are connected to an upsampling block, and each group of decoding modules is composed of a fifth convolutional layer, a sixth convolutional layer, a short subtraction convolution block, a middle subtraction convolution block, a long subtraction convolution block, and a merging layer; The eye movement image is input into an encoder of a pre-trained segmentation model to extract image features, and the decoder is used to receive the image features extracted by the encoder, thereby generating a target area included between the upper and lower eyelids in the eye movement image.

27. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting the palpebral fissure based on a texture-aware neural network described in any one of claims 1 to 24 is implemented.

Images