Recognition method for recognition of tissue lesions based on complementary attention mechanism

Abstract

The disclosure describes a recognition method for recognizing tissue lesions based on a complementary attention mechanism, comprising: acquiring a tissue image; an artificial neural network module comprising a first artificial neural network, a second artificial neural network and a third artificial neural network, the first artificial neural network being configured to be capable of performing feature extraction on the tissue image to obtain a feature map, the second artificial neural network being configured to be capable of obtaining an attention heat map indicating a lesion area, the third artificial neural network being configured to be capable of recognizing the tissue image based on the feature map, using the third artificial neural network to recognize the inspection image based on the feature map and the complementary attention heat map to obtain a third recognition result, combining the third recognition result with a labeled image having a label result of no lesion to obtain a third loss function when using the complementary attention mechanism, and optimizing the artificial neural network module using the third loss function, thereby effectively improving the recognition rate of tissue lesions.

Description

[0001] The present application is a divisional application of a patent application with application number 202011364685X, titled "Recognition method and recognition system for recognition of tissue lesion based on artificial neural network", and filed on November 27, 2020. TECHNICAL FIELD

[0002] The present disclosure relates generally to a recognition method for recognition of tissue lesion based on complementary attention mechanism. BACKGROUND

[0003] With the development and maturity of artificial intelligence technology, artificial intelligence technology has been gradually popularized in various aspects of the medical field. In particular, medical imaging in medicine is currently a popular field of application of artificial intelligence technology. Medical imaging is a useful tool for diagnosing many diseases, and a large amount of medical image data is generated during the medical imaging process. Processing and recognizing these image data requires a large amount of time for physicians, and it is difficult to ensure the accuracy of the recognition. In medical images, artificial intelligence technology is mainly used to recognize tissue lesions in the image to improve the accuracy of tissue lesion recognition.

[0004] At present, convolutional neural network (CNN) is usually used to recognize medical images by applying artificial intelligence technology. The convolution structure of convolutional neural network can reduce the amount of memory occupied by deep network, and has three key operations, one of which is local receptive field, the second is weight sharing, and the third is pooling layer. Thus, the number of network parameters can be effectively reduced, and the overfitting problem of convolutional neural network can be alleviated. The structure of convolutional neural network can better adapt to the structure of medical images and extract and recognize features.

[0005] However, for some lesion sites such as fundus lesion sites, the lesion area is small and irregularly distributed. Generally, the convolutional neural network applying attention mechanism tends to ignore the lesion area with low attention in the attention heat map, resulting in misjudgment phenomenon, thereby reducing the accuracy of tissue lesion recognition of these lesion areas. SUMMARY

[0006] The present disclosure is completed in view of the above-mentioned prior art, and aims to provide a recognition method and recognition system for recognition of tissue lesion based on artificial neural network, which can effectively improve the accuracy of tissue lesion recognition.

[0007] To this end, the first aspect of the present disclosure provides a recognition method for tissue lesion recognition based on an artificial neural network, which comprises: acquiring a tissue image, the tissue image being a tissue image acquired by an acquisition device; receiving the tissue image and performing lesion recognition on the tissue image by using an artificial neural network module, the artificial neural network module comprising a first artificial neural network, a second artificial neural network, and a third artificial neural network, the first artificial neural network being configured to be capable of performing feature extraction on the tissue image to obtain a feature map, the second artificial neural network being configured to be capable of obtaining an attention heat map indicating a lesion area, the third artificial neural network being configured to be capable of recognizing the tissue image based on the feature map, a training step of the artificial neural network module comprising: preparing a training data set, the training data set comprising a plurality of examination images and a labeled image associated with the examination images, the labeled image comprising a labeled result of having a lesion or a labeled result of having no lesion, performing feature extraction on the examination images by using the first artificial neural network to obtain a feature map, obtaining an attention heat map indicating a lesion area and a complementary attention heat map indicating a non-lesion area by using the second artificial neural network, the examination image being composed of the lesion area and the non-lesion area, recognizing the examination image based on the feature map by using the third artificial neural network to obtain a first recognition result, recognizing the examination image based on the feature map and the attention heat map by using the third artificial neural network to obtain a second recognition result, recognizing the examination image based on the feature map and the complementary attention heat map by using the third artificial neural network to obtain a third recognition result, combining the first recognition result with the labeled image to obtain a first loss function when the attention mechanism is not used, combining the second recognition result with the labeled image to obtain a second loss function when the attention mechanism is used, combining the third recognition result with the labeled image having the labeled result of having no lesion to obtain a third loss function when the complementary attention mechanism is used, obtaining a total loss function comprising a first loss term based on the first loss function, a second loss term based on a difference between the second loss function and the first loss function, and a third loss term based on the third loss function by using the first loss function, the second loss function, and the third loss function, and optimizing the artificial neural network module by using the total loss function. In this case, the recognition result of the tissue lesion recognition can be obtained by using the artificial neural network module, and the artificial neural network module can be optimized by using the total loss function, so that the accuracy of the tissue lesion recognition can be improved.

[0008] In addition, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the total loss function further comprises a total area term of the attention heat map, and the total area term is used to evaluate the area of the lesion region. In this case, the area of the lesion region in the attention heat map can be evaluated by using the fifth loss term, and the number of pixels in the attention heat map that have a greater impact on the recognition result can be controlled, so that the attention of the network is limited to the pixels that have a greater impact on the recognition result.

[0009] In addition, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the total loss function further comprises a regularization term for the attention heat map. In this case, overfitting of the artificial neural network module can be inhibited.

[0010] In addition, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the first artificial neural network, the second artificial neural network and the third artificial neural network are trained simultaneously. In this case, the training speed can be accelerated.

[0011] In addition, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the third artificial neural network comprises an input layer, an intermediate layer and an output layer connected in sequence, and the output layer is configured to output the recognition result reflecting the inspection image. In this case, the third artificial neural network can be used to output the recognition result reflecting the inspection image.

[0012] In addition, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the training mode of the artificial neural network module is weak supervision. In this case, the artificial neural network module can be used to obtain a recognition result with a large amount of information from a label result with a small amount of information.

[0013] In addition, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the first loss function is used to evaluate the degree of inconsistency between the recognition result of the inspection image when the attention mechanism is not used and the label result. In this case, the accuracy of the artificial neural network module in recognizing the tissue lesion when the attention mechanism is not used can be improved.

[0014] Additionally, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the second loss function is used to evaluate the degree of inconsistency between the recognition result of the examination image when using the attention mechanism and the annotation result. In this case, the accuracy of the tissue lesion recognition of the artificial neural network module when using the attention mechanism can be improved.

[0015] Additionally, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the third loss function is used to evaluate the degree of inconsistency between the recognition result of the examination image when using the complementary attention mechanism and the annotation result of no lesion. In this case, the accuracy of the tissue lesion recognition of the artificial neural network module when using the complementary attention mechanism can be improved.

[0016] Additionally, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the artificial neural network module is optimized by using the total loss function to minimize the total loss function. In this case, the total loss function can be minimized to improve the accuracy of the tissue lesion recognition of the artificial neural network module.

[0017] Additionally, in the recognition method for recognizing the tissue lesion based on the artificial neural network according to the first aspect of the present disclosure, optionally, the tissue lesion is a fundus lesion. In this case, the recognition result of the fundus image about the fundus lesion can be obtained by using the artificial neural network module.

[0018] The second aspect of the present disclosure provides a recognition system for recognizing the tissue lesion based on the artificial neural network, which is characterized in that the recognition method provided in the first aspect of the present disclosure is used to recognize the tissue lesion. In this case, the recognition system can be used to recognize the tissue lesion of the tissue image.

[0019] According to the present disclosure, a recognition method and a recognition system for recognizing the tissue lesion based on the artificial neural network can be provided, which can effectively improve the accuracy of recognizing the tissue lesion. BRIEF DESCRIPTION OF DRAWINGS

[0020] Embodiments of the present disclosure will now be explained in further detail by way of example only with reference to the drawings, in which:

[0021] Figure 1 is a schematic diagram of an electronic device according to an example of the present disclosure.

[0022] Figure 2 is a tissue image according to an example of the present disclosure.

[0023] Figure 3is a structural block diagram showing a recognition system related to the artificial neural network-based tissue lesion recognition of the example of the present disclosure.

[0024] Figure 4 is a block diagram showing one example of the artificial neural network module related to the example of the present disclosure.

[0025] Figure 5 is a block diagram showing a modification of the artificial neural network module related to the example of the present disclosure.

[0026] Figure 6 is a structural schematic diagram showing the first artificial neural network related to the example of the present disclosure.

[0027] Figure 7 is a structural block diagram showing a training system related to the artificial neural network-based tissue lesion recognition of the example of the present disclosure.

[0028] Figure 8 is a flowchart showing a training method related to the artificial neural network-based tissue lesion recognition of the example of the present disclosure.

[0029] Fig. 9(a) is a schematic diagram showing one example of a fundus image trained without using an attention mechanism related to the example of the present disclosure.

[0030] Fig. 9(b) is a schematic diagram showing one example of a lesion region of a fundus image trained using a complementary attention mechanism related to the example of the present disclosure.

[0031] Main reference numerals: 1… electronic device, 10… processor, 20… memory, 30… computer program, 40… recognition system, 410… acquisition module, 4200… backbone neural network, 420… artificial neural network module, 421… first artificial neural network, 422… second artificial neural network, 423… third artificial neural network, 424… feature combination module, 430… training system, 431… storage module, 432… processing module, 433… optimization module, C1… first convolution layer, C2… second convolution layer, C3… third convolution layer, S1… first pooling layer, S2… second pooling layer, S3… third pooling layer DETAILED DESCRIPTION

[0032] Hereinafter, the preferred embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same parts are given the same symbols, and repeated description is omitted. In addition, the drawings are merely schematic diagrams, and the proportions of the sizes of the components with respect to each other or the shapes of the components, etc. can be different from actual ones.

[0033] Figure 1 is a schematic diagram showing an electronic device related to the embodiment of the present disclosure.

[0034] like Figure 1 As shown, the identification system 40 for tissue lesion identification based on artificial neural networks disclosed herein can be carried by an electronic device 1 (such as a computer). In some examples, the electronic device 1 may include one or more processors 10, a memory 20, and a computer program 30 arranged in the memory 20. The one or more processors 10 may include a central processing unit, a graphics processing unit, and any other electronic components capable of processing data. For example, the processor 10 may execute instructions stored in the memory 20.

[0035] In some examples, memory 20 may be a computer-readable medium capable of carrying or storing data. In some examples, memory 20 may include, but is not limited to, non-volatile memory or flash memory. In some examples, memory 20 may also be, for example, ferroelectric random access memory (FeRAM), magnetic random access memory (MRAM), phase-change random access memory (PRAM), or resistive random access memory (RRAM). This reduces the possibility of data loss due to sudden power outages.

[0036] In other examples, memory 20 may also be other types of readable storage media, such as read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), and compact disc read-only memory (CD-ROM).

[0037] In some examples, memory 20 can be optical disc storage, magnetic disk storage, or magnetic tape storage. Therefore, a suitable memory 20 can be selected according to different circumstances.

[0038] In some examples, the computer program 30 can include instructions executed by the one or more processors 10, by executing the instructions, the identification system 40 can perform the tissue lesion identification on the tissue image. In some examples, the computer program 30 can be deployed in a local computer, or can be deployed in a server in the cloud.

[0039] In some examples, the computer program 30 can be stored in a computer readable medium. The computer readable storage medium can include one or more of a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0040] Figure 2 is a tissue image related to an example of the present disclosure. Figure 3 is a structural block diagram of an identification system 40 for tissue lesion identification based on artificial neural network related to an example of the present disclosure.

[0041] In some examples, the identification system 40 for tissue lesion identification based on artificial neural network can be used to perform the tissue lesion identification on the tissue image, and obtain the identification result. In some examples, the identification system 40 for tissue lesion identification can also be referred to as the identification system 40.

[0042] In some examples, as shown in Figure 3 the identification system 40 can include an acquisition module 410, an artificial neural network module 420, and a training system 430 for tissue lesion identification based on artificial neural network. In some examples, the acquisition module 410 can be used to acquire the tissue image. In some examples, the artificial neural network module 420 can be used to perform feature extraction, tissue lesion identification, and other processing on the tissue image, and obtain the identification result of the tissue lesion identification. In some examples, the training system 430 for tissue lesion identification based on artificial neural network can be used to train the artificial neural network module 420. In some examples, the training system 430 can use the first identification result, the second identification result, and the third identification result obtained by the artificial neural network module 420, and obtain the total loss function based on the first identification result, the second identification result, and the third identification result to optimize the artificial neural network module 420. In this case, the first identification result, the second identification result, and the third identification result can be obtained, and the total loss function can be obtained based on the first identification result, the second identification result, and the third identification result, so that the artificial neural network module 420 can be optimized using the total loss function, and the accuracy of the tissue lesion identification of the artificial neural network module 420 can be improved.

[0043] In some examples, the training system 430 for artificial neural network based tissue lesion recognition can also be referred to as the training system 430.

[0044] In some examples, the recognition system 40 can further comprise a pre-processing module and a judgment module (not shown).

[0045] In some examples, the tissue image can be an image of a tissue cavity taken from a CT scan, a PET-CT scan, a SPECT scan, an MRI, an ultrasound, an X-ray, a mammogram, an angiogram, a fluorogram, a capsule endoscopy, or a combination thereof. In some examples, the tissue image can be acquired by the acquisition module 410.

[0046] In some examples, the acquisition module 410 can be configured to acquire the tissue image, which can be acquired by a camera, an ultrasound imager, or an X-ray scanner, etc.

[0047] In some examples, the tissue image can be, for example, a fundus image, an esophagus image, a stomach image, a large intestine image, a colon image, or a small intestine image. As Figure 2 shown, the tissue image can be a fundus image. In this case, the fundus lesion recognition can be performed on the fundus image by the recognition system 40.

[0048] In some examples, the tissue lesion recognition can be to recognize a tissue lesion in the tissue image to obtain a recognition result.

[0049] In some examples, in the case where the tissue image is a fundus image, the tissue lesion can be a fundus lesion. In this case, the recognition result of the fundus image regarding the fundus lesion can be obtained by the artificial neural network module 420.

[0050] In some examples, the tissue image can be composed of a lesion area and a non-lesion area.

[0051] In some examples, the tissue image (color image) with a tissue lesion generally contains obvious features such as erythema and swelling, and therefore, these features can be automatically extracted and recognized by the trained artificial neural network to help the patient identify possible lesions. In this way, the accuracy and speed of recognition can be improved, and the problems of large errors and long time consumption of human doctors in reading films one by one with their own experience can be reduced.

[0052] In some examples, in the case where the tissue image is a fundus image, the tissue image can be classified according to functions. For example, in the training step, the tissue image can be an examination image, a labeled image (described later).

[0053] In some examples, the image inputted into the artificial neural network module 420 can be a tissue image. In this case, the tissue image can be subjected to tissue lesion recognition by the artificial neural network module 420.

[0054] In some examples, the recognition system 40 can be used for tissue lesion recognition of a tissue image. In some examples, when a tissue image enters the recognition system 40, the tissue image can be subjected to pre-processing, feature extraction, tissue lesion recognition, and the like.

[0055] In some examples, the recognition system 40 can further include a pre-processing module and a judgment module. The pre-processing module can be used to pre-process a tissue image and input the pre-processed tissue image into the artificial neural network module 420.

[0056] In some examples, the pre-processing module can pre-process a tissue image. In some examples, the pre-processing can include at least one of region of interest detection, image cropping, size adjustment, and normalization. In this case, the subsequent tissue lesion recognition and judgment of the tissue image by the artificial neural network module 420 can be facilitated. In some examples, the tissue image can be, for example, a fundus image, an esophagus image, a stomach image, a large intestine image, a colon image, or a small intestine image.

[0057] In some examples, the pre-processing module can include a region detection unit, an adjustment unit, and a normalization unit.

[0058] In some examples, the region detection unit can detect a region of interest from a tissue image. For example, if the tissue image is a fundus image, a fundus region centered on the optic disc, or a fundus region containing the optic disc and centered on the macular center can be detected from the fundus image. In some examples, the region detection unit can detect a region of interest in a tissue image by, for example, a sampling threshold method, a Hough transformation, and the like.

[0059] In some examples, the adjustment unit can be used to crop and adjust the size of a tissue image. Due to differences in devices used to acquire tissue images or differences in shooting conditions, the obtained tissue images can differ in resolution, size, and the like. In this case, the tissue images can be cropped and adjusted in size to reduce the differences. In some examples, a tissue image can be cropped according to a specific shape. In some examples, the specific shape can include, but is not limited to, a square, a rectangle, a circle, an ellipse, and the like.

[0060] In other examples, the size of the tissue images can be adjusted to a prescribed size by the adjusting unit. For example, the prescribed size can be 256x256, 512x512, or 1024x1024, etc. However, examples of the present disclosure are not limited thereto, and in other examples, the size of the tissue images can also be any other size of a specified format. For example, the size of the tissue images can be 128x128, 768x768, or 2048x2048, etc.

[0061] In some examples, the preprocessing module can include a normalization unit. The normalization unit can be configured to normalize a plurality of tissue images.

[0062] In some examples, the manner of normalization of the normalization unit is not particularly limited, and for example, zero mean, unit standard deviation, etc. can be used. In addition, in some examples, normalization in the range of [0, 1] can also be used. In this case, by normalization, the difference between different tissue images can be overcome.

[0063] In some examples, the normalization includes normalization of image format, image slice interval, image intensity, image contract, and image orientation. In some examples, the tissue images can be normalized to DICOM format, NIfTI format, or raw binary format.

[0064] Figure 4 is a block diagram illustrating an example of an artificial neural network module involved in examples of the present disclosure.

[0065] As described above, the recognition system 40 can include an artificial neural network module 420. In some examples, the artificial neural network module 420 can be configured to perform tissue lesion recognition on the tissue images. In some examples, the artificial neural network module 420 can include a plurality of artificial neural networks. In some examples, the artificial neural networks can be trained using one or more processors 10. Generally, an artificial neural network can include artificial neurons or nodes that can be configured to receive tissue images and perform operations on the tissue images based on weights, and then selectively pass the results of the operations to other neurons or nodes. The weights can be associated with the artificial neurons or nodes and at the same time constrain the output of other artificial neurons. The weights (i.e., network parameters) can be determined by iteratively training the artificial neural network using a training dataset (described later).

[0066] In some examples, as shown in Figure 4 the artificial neural network module 420 can include a backbone neural network 4200 and a second artificial neural network 422.

[0067] In some examples, the backbone neural network 4200 can include a first artificial neural network 421, a third artificial neural network 423, and a feature combination module 424.

[0068] In some examples, the first artificial neural network 421 can receive the tissue image and perform feature extraction on the tissue image to obtain a feature map.

[0069] In some examples, the second artificial neural network 422 can receive the feature map and the identification result from the third artificial neural network 423 and obtain an attention heat map indicating a lesion region and a complementary attention heat map indicating a non-lesion region. It is to be noted that in other examples, the above-mentioned attention heat map or complementary attention heat map can also be considered as a kind of feature map.

[0070] In some examples, the feature combination module 424 can receive the feature map, the attention heat map, and the complementary attention heat map and output a feature combination set. In some examples, the feature combination module 424 can also directly output the feature map.

[0071] In some examples, the third artificial neural network 423 can receive the feature map or the feature combination set and output an identification result of the tissue lesion identification of the tissue image.

[0072] In some examples, the tissue image (e.g., the pre-processed tissue image) input into the artificial neural network module 420 can enter the first artificial neural network 421 and finally output the identification result by the third artificial neural network 423.

[0073] Figure 5 is a block diagram showing a variant of the artificial neural network module involved in the examples of the present disclosure.

[0074] In addition, in some examples, as shown in Figure 5 the artificial neural network module 420 can include a backbone neural network 4200 and a second artificial neural network 422.

[0075] In some examples, as shown in Figure 5 the backbone neural network 4200 can include a first artificial neural network 421 and a third artificial neural network 423.

[0076] In some examples, the third artificial neural network 423 can have a feature combination function. For details, please refer to the related description in the feature combination module 424.

[0077] In some examples, the first artificial neural network 421 can receive the tissue image and perform feature extraction on the tissue image to obtain a feature map.

[0078] In some examples, the second artificial neural network 422 can further obtain a complementary attention heat map indicating non-lesion regions from the attention heat map.

[0079] In some examples, the third artificial neural network 423 can receive the feature map, the attention heat map and the complementary attention heat map and output a recognition result of the tissue lesion recognition of the tissue image. In some examples, the attention heat map can be a heat map indicating lesion regions obtained based on an attention mechanism. In some examples, the attention heat map can show the importance of each pixel point in the tissue image when forming the feature map.

[0080] In some examples, the complementary attention heat map can be a heat map indicating non-lesion regions obtained based on a complementary attention mechanism.

[0081] In some examples, the complementary attention heat map can be a complementary image of the attention heat map. In some examples, the size and format of the complementary attention heat map can be the same as the size and format of the attention heat map.

[0082] As described above, the artificial neural network module 420 can include a first artificial neural network 421 (see Figure 5 ).

[0083] In some examples, the first artificial neural network 421 can use one or more deep neural networks to automatically identify features in the tissue image.

[0084] In some examples, the first artificial neural network 421 can be used to receive the tissue image preprocessed by the preprocessing module and generate one or more feature maps. In some examples, the first artificial neural network 421 can combine multiple layers of low-level features (pixel-level features), for example. In this case, an abstract description of the tissue image can be achieved.

[0085] In some examples, the first artificial neural network 421 can include an input layer, an intermediate layer and an output layer connected in sequence. The input layer can be configured to receive the tissue image preprocessed by the preprocessing module. The intermediate layer can be configured to extract a feature map based on the tissue image, and the output layer can be configured to output the feature map.

[0086] In some examples, the tissue image input into the artificial neural network module 420 can be converted into a pixel matrix, which can be a three-dimensional pixel matrix, for example. The length and width of the three-dimensional matrix can represent the size of the image, and the depth of the three-dimensional matrix represents the color channel of the image. In some examples, the depth can be 1 (i.e., the tissue image is a grayscale image), and in some examples, the depth can be 3 (i.e., the tissue image is a color image in RGB color mode).

[0087] In some examples, the first artificial neural network 421 can be a convolutional neural network. Because convolutional neural networks have advantages such as local receptive fields and weight sharing, they can significantly reduce parameter training, thus improving processing speed and saving hardware costs. Furthermore, convolutional neural networks can more effectively identify tissue images.

[0088] Figure 6 This is a schematic diagram illustrating the structure of the first artificial neural network 421 involved in the example of this disclosure.

[0089] In some examples, the first artificial neural network 421 may contain multiple intermediate layers, each of which may include multiple neurons or nodes. Each neuron or node in the intermediate layer may have an activation function (such as the ReLU (rectified linear unit) function, the sigmoid function, or the tanh function) applied to its output. The activation functions applied to different neurons affect the activation functions applied to other neurons.

[0090] In some examples, such as Figure 6 As shown, the intermediate layers of the first artificial neural network 421 may include multiple convolutional layers and multiple pooling layers. In some examples, convolutional layers and pooling layers may be combined alternately. In some examples, the tissue image may be passed sequentially through a first convolutional layer C1, a first pooling layer S1, a second convolutional layer C2, a second pooling layer S2, a third convolutional layer C3, and a third pooling layer S3. In this case, convolutional and pooling processing of the tissue image can be performed alternately.

[0091] In other examples, the first artificial neural network 421 may not include a pooling layer, thereby avoiding data loss during pooling and simplifying the network structure.

[0092] In some examples, convolutional layers can use convolutional kernels to convolve tissue images within a convolutional neural network. In this case, it is possible to obtain more abstract features, allowing the matrix depth to become deeper.

[0093] In some examples, the kernel size can be 3x3. In other examples, the kernel size can be 5x5. In some examples, a 5x5 kernel can be used in the first convolutional layer C1, while other convolutional layers use 3x3 kernels. This improves training efficiency. In some examples, the kernel size can be set arbitrarily. In this case, the kernel size can be chosen based on the image size and computational cost.

[0094] In some examples, the pooling layer can also be referred to as a down-sampling layer. In some examples, the input tissue image can be processed using a pooling manner such as max-pooling, mean-pooling or stochastic-pooling. In this case, on the one hand, the feature dimension can be reduced to improve the operation efficiency, and on the other hand, the convolutional neural network can extract more abstract high-level features to improve the accuracy of identifying the tissue lesions.

[0095] In addition, in some examples, the number of convolution layers and the number of pooling layers in the above-mentioned convolutional neural network can also be increased according to the situation. In this case, the convolutional neural network can extract more abstract high-level features to further improve the accuracy of identifying the tissue lesions.

[0096] In some examples, the pre-processed tissue image can output a feature map corresponding to the tissue image after passing through the first artificial neural network 421. In some examples, the feature map can have multiple depths. In some examples, the pre-processed tissue image can output multiple feature maps after passing through the first artificial neural network 421. In some examples, the multiple feature maps can correspond to different features respectively. In some examples, the tissue lesions can be identified based on the features corresponding to the feature maps.

[0097] In some examples, the feature map can be sequentially subjected to de-convolution and up-sampling processing before the first artificial neural network 421 outputs the feature map. In some examples, the feature map can be subjected to multiple de-convolution and up-sampling processing. For example, the feature map can sequentially pass through a first de-convolution layer, a first up-sampling layer, a second de-convolution layer, a second up-sampling layer, a third de-convolution layer and a third up-sampling layer. In this case, the size of the feature map can be changed and part of the data information of the tissue image can be retained.

[0098] In some examples, the number of de-convolution layers can be the same as the number of convolution layers, and the number of pooling layers (down-sampling layers) can be the same as the number of up-sampling layers. In this way, the size of the feature map can be the same as that of the tissue image.

[0099] In some examples, the tissue image processed by the convolution layer (pooling layer) can be selected for convolution before the feature map passes through the de-convolution layer (up-sampling). For example, the feature map can be subjected to convolution processing with the output image of the second convolution layer C2 (the second pooling layer S2) before the feature map enters the second de-convolution layer (the second up-sampling layer). The feature map can be subjected to convolution processing with the output image of the first convolution layer C1 (the first pooling layer S1) before the feature map enters the third de-convolution layer (the third up-sampling layer). In this case, the data information lost during the processing of the pooling layer or the convolution layer can be supplemented.

[0100] In some examples, after generating the feature map by the first artificial neural network 421, an attention heat map matching the feature map can be generated by the second artificial neural network 422.

[0101] In the present embodiment, the second artificial neural network 422 is an artificial neural network with an attention mechanism. In some examples, the output image of the second artificial neural network 422 can include an attention heat map and a complementary attention heat map.

[0102] In some examples, the second artificial neural network 422 can include an input layer, an intermediate layer and an output layer connected in sequence. The input layer is configured to receive the feature map and the partial weight of the third artificial neural network 423 or the recognition result of the tissue lesion recognition. The intermediate layer is configured to obtain the feature weight based on the feature map and the partial weight of the third artificial neural network 423 or the recognition result of the tissue lesion recognition. The intermediate layer is configured to generate the attention heat map and / or the complementary attention heat map based on the feature map and the feature weight. The output layer is configured to output the attention heat map and / or the complementary attention heat map. In some examples, the feature map can be generated by the first artificial neural network 421.

[0103] In some examples, the attention mechanism can selectively filter out a small amount of important information from a large amount of information in the input feature map and focus on the important information.

[0104] In some examples, the attention heat map can be an image representing attention in the form of a heat map. Generally speaking, the pixels in the corresponding position of the attention heat map in red or white have a greater impact on the tissue image tissue lesion recognition. The pixels in the corresponding position of the attention heat map in blue or black have a smaller impact on the tissue image tissue lesion recognition.

[0105] In some examples, each feature map can be weighted by the feature weight, and the attention heat map can be obtained. In some examples, the feature weight can be obtained by the attention mechanism. In some examples, the attention mechanism can include, but is not limited to, a channel attention mechanism (CAM), a gradient-based channel attention mechanism (Grad-CAM), a gradient-based reinforced channel attention mechanism (Grad-CAM++), a spatial attention mechanism (SAM), etc.

[0106] In some examples, the third artificial neural network 423 has a global pooling layer and a fully connected layer. In some examples, the feature weight can be the weight in the third artificial neural network 423 from the fully connected layer to the output layer of the third artificial neural network 423. For example, for the case that the tissue image is a fundus image, the third artificial neural network 423 can receive the feature map of the fundus image and obtain a first recognition result (described later). If the first recognition result is "macula", the weight from each neuron or node of the global pooling layer to the recognition result "macula" in the fully connected layer is extracted as the feature weight.

[0107] In some examples, the feature weight can be calculated based on the tissue lesion recognition result in the third artificial neural network 423. In some examples, the partial derivative of the first recognition result (e.g. the probability of the tissue lesion) of the third artificial neural network 423 to all pixels in a feature map can be calculated, and the partial derivative of all pixels in the feature map is globally pooled to obtain the feature weight corresponding to the feature map.

[0108] In some examples, the attention heat map matching the feature map can be generated by the second artificial neural network 422. In some examples, the complementary attention heat map can be generated by the second artificial neural network 422. In some examples, the two pixel values corresponding to the same pixel position in the attention heat map and the complementary attention heat map are inversely related. In some examples, the attention heat map and / or the complementary attention heat map can be normalized. In some examples, the sum or product of the two pixel values corresponding to the same pixel position in the attention heat map and the complementary attention heat map is a constant value.

[0109] In some examples, the total variation can be used to regularize the attention heat map and / or the complementary attention heat map.

[0110] In some examples, the output layer of the first artificial neural network 421 and the second artificial neural network 422 can be connected to the feature combination module 424.

[0111] In some examples, the feature combination module 424 can have an input layer and an output layer. In some examples, the output layer of the feature combination module 424 can be a feature map or a feature combination set. In some examples, the input layer of the feature combination module 424 can receive a feature map, an attention heat map or a complementary attention heat map.

[0112] In some examples, the feature combination module 424 can combine the feature map output by the first artificial neural network 421 and the attention heat map or the complementary attention heat map output by the second artificial neural network 422 to form a feature combination set.

[0113] In some examples, the set of feature combinations can include at least one of the first set of feature combinations and the second set of feature combinations.

[0114] In some examples, the feature combination module 424 can perform feature combination on the feature map output by the first artificial neural network 421 and the attention heat map output by the second artificial neural network 422 to form the first set of feature combinations.

[0115] In some examples, the feature combination module 424 can perform feature combination on the feature map output by the first artificial neural network 421 and the complementary attention heat map output by the second artificial neural network 422 to form the second set of feature combinations.

[0116] In some examples, the feature combination module 424 can directly output the feature map.

[0117] In some examples, the feature combination module 424 can also calculate the difference between the feature map and the attention heat map to obtain the first set of feature combinations.

[0118] In some examples, the feature combination module 424 can also calculate the difference between the feature map and the complementary attention heat map to obtain the second set of feature combinations.

[0119] In some examples, the feature combination module 424 can also calculate the convolution between the feature map and the attention heat map to obtain the first set of feature combinations.

[0120] In some examples, the feature combination module 424 can also calculate the convolution between the feature map and the complementary attention heat map to obtain the second set of feature combinations.

[0121] In some examples, the feature combination module 424 can also calculate the mean between the feature map and the attention heat map to obtain the first set of feature combinations.

[0122] In some examples, the feature combination module 424 can also calculate the mean between the feature map and the complementary attention heat map to obtain the second set of feature combinations.

[0123] In addition, in other examples, the feature combination module 424 can perform linear or nonlinear transformation on the feature map and the attention heat map to obtain the first set of feature combinations.

[0124] In addition, in other examples, the feature combination module 424 can perform linear or nonlinear transformation on the feature map and the complementary attention heat map to obtain the second set of feature combinations.

[0125] In some examples, the output layer of the feature combination module 424 can output the feature map, the first feature combination set, and the second feature combination set. In some examples, the feature map, the first feature combination set, and the second feature combination set outputted by the feature combination module 424 can be inputted into the third artificial neural network 423 and be organized by the third artificial neural network 423 for lesion recognition.

[0126] In some examples, the feature combination module 424 can be incorporated into the third artificial neural network 423 and be a part of the third artificial neural network 423. In this case, the artificial neural network module 420 can include the first artificial neural network 421, the second artificial neural network 422, and the third artificial neural network 423.

[0127] In some examples, in the case that the feature combination module 424 is incorporated into the third artificial neural network 423, the input layer of the third artificial neural network 423 can receive the feature map, the attention heat map, or the complementary attention heat map.

[0128] In some examples, the third artificial neural network 423 can include an input layer, an intermediate layer, and an output layer connected in sequence. In some examples, the output layer can be configured to output a result reflecting the recognition of the tissue image. In this case, the third artificial neural network 423 can be used to output a result reflecting the recognition of the tissue image. In some examples, the output layer of the third artificial neural network 423 can include a Softmax layer. In some examples, the intermediate layer of the third artificial neural network 423 can be a fully connected layer.

[0129] In some examples, the final classification can be performed by the fully connected layer, and the probability of the tissue image belonging to each category of the tissue lesion can be finally obtained through the Softmax layer. In this case, the recognition result of the tissue lesion recognition of the tissue image can be obtained based on the probability.

[0130] In some examples, the third artificial neural network 423 can include various linear classifiers, such as a single-layer fully connected layer.

[0131] In some examples, the third artificial neural network 423 can include various nonlinear classifiers. For example, Logistic Regression, Random Forest, or Support Vector Machines, etc.

[0132] In some examples, the third artificial neural network 423 can include multiple classifiers. In some examples, the classifiers can give recognition results of the tissue lesion recognition of the tissue image. For example, in the case that the tissue image is a fundus image, recognition results of the fundus lesion recognition of the fundus image can be given. In this case, the fundus lesion recognition can be performed on the fundus image.

[0133] In some examples, the output of the third neural network 423 can be values between 0 and 1, which can be used to represent the probability that the tissue image belongs to each tissue lesion category.

[0134] In some examples, when the probability that the tissue image belongs to a certain tissue lesion category is the highest, the category is taken as the recognition result of the tissue lesion recognition of the tissue image. For example, in the probability that the tissue image belongs to each tissue lesion category, if the probability of the category of no lesion is the highest, the recognition result of the tissue lesion recognition of the tissue image can be no lesion. For example, in the process of performing fundus lesion recognition on a fundus image, the prediction probabilities of macular and no lesion output by the third artificial neural network 423 are 0.8 and 0.2 respectively, and it can be considered that the fundus image has macular lesion.

[0135] In some examples, the third artificial neural network 423 can output recognition results matched with the tissue image. In some examples, the recognition results can include the first recognition result when the attention mechanism is not used, the second recognition result when the attention mechanism is used, and the third recognition result when the attention mechanism and the complementary attention are used.

[0136] In some examples, the third artificial neural network 423 can perform tissue lesion recognition on the feature map of the output of the feature combination module 424 and obtain the first recognition result.

[0137] In some examples, the third artificial neural network 423 can perform tissue lesion recognition on the first feature combination set of the output of the feature combination module 424 and obtain the second recognition result.

[0138] In some examples, the third artificial neural network 423 can perform tissue lesion recognition on the second feature combination set of the output of the feature combination module 424 and obtain the third recognition result.

[0139] In some examples, the identification result can include both a lesion and no lesion. In some examples, the identification result can also include no lesion or a specific lesion type. For example, in the case of the tissue image being a fundus image, the identification result can include, but is not limited to, one of no lesion, hypertensive retinopathy, or diabetic retinopathy. In this case, the identification result of the fundus lesion identification of the fundus image can be obtained. In some examples, the identification result of a tissue image can be multiple. For example, the identification result can be both hypertensive retinopathy and diabetic retinopathy.

[0140] In some examples, the identification system 40 can further include a judgment module.

[0141] In some examples, the judgment module can receive the output of the artificial neural network module 420. In this case, the output result of the artificial neural network module 420 can be synthesized by the judgment module and the final identification result can be output, so that a summary report can be generated.

[0142] In some examples, the first identification result can be taken as the final identification result of the tissue image. In this case, when the tissue lesion identification of the tissue image is performed by using the artificial neural network module 420, the tissue lesion identification of the tissue image can be performed by the backbone neural network 4200 including the first artificial neural network 421 and the third artificial neural network 423, so as to speed up the identification.

[0143] In some examples, the second identification result can be taken as the final identification result of the tissue image.

[0144] As described above, the third identification result can be obtained based on the complementary attention mechanism. In some examples, the final identification result of the tissue image can be obtained based on the first identification result, the second identification result, and the third identification result. For example, in some examples, the final identification result can include the second identification result and the third identification result. In some examples, the final identification result can include the first identification result and the third identification result.

[0145] In some examples, the summary report generated by the judgment module can include at least one of the first identification result, the second identification result, the third identification result, and the final identification result. In some examples, the judgment module can generate a lesion indication map by color coding the tissue image based on the attention heat map to indicate the lesion area. The summary report generated by the judgment module can include the lesion indication map.

[0146] In some examples, the summary report generated by the judgment module can include the position of the corresponding lesion, and the position can be marked by using a marker box.

[0147] In some examples, the summary report generated by the assessment module can display lesion areas in tissue images as heatmaps. Specifically, in the heatmap, areas with a high probability of lesion can be displayed in red or white, while areas with a low probability of lesion can be displayed in blue or black. In this case, lesion areas can be indicated in a visually appealing way.

[0148] In some examples, the judgment module can also be used to select the lesion area. In some examples, the lesion area can be selected using a fixed shape (such as a triangle, circle, quadrilateral, or other regular shapes). In some examples, the lesion area can also be outlined. In this case, the lesion area can be displayed visually.

[0149] In some examples, the judgment module can also be used to delineate lesion areas. For instance, it can analyze the values ​​corresponding to each pixel in the attention heatmap and classify pixels with values ​​greater than a first preset value as lesion areas, and pixels with values ​​less than the first preset value as non-lesion areas.

[0150] The identification method for tissue lesion identification based on artificial neural networks disclosed herein is implemented by identification system 40.

[0151] In some examples, the recognition method includes acquiring tissue images and using the artificial neural network module 420 to obtain recognition results for tissue lesion identification. In some examples, the tissue images can be tissue images acquired through an acquisition device. In some examples, the artificial neural network module 420 is trained using a training system 430. In this case, the recognition results for tissue lesion identification can be obtained using the artificial neural network module 420, and the artificial neural network module 420 can be optimized using a total loss function, thereby improving the accuracy of tissue lesion identification.

[0152] The following describes in detail, with reference to the accompanying drawings, the training method (sometimes referred to simply as the training method) and training system for tissue lesion recognition based on artificial neural networks involved in this embodiment.

[0153] In some examples, the training method can be implemented using a training system 430 for tissue lesion recognition based on an artificial neural network. In this case, the artificial neural network module 420 can be trained using the training system 430.

[0154] Figure 7 This is a block diagram illustrating the structure of a training system 430 for tissue lesion recognition based on an artificial neural network, as described in this disclosure example.

[0155] In some examples, such as Figure 7As shown, the training system 430 can include a storage module 431, a processing module 432, and an optimization module 433. In some examples, the storage module 431 can be configured to store the training dataset. In some examples, the processing module 432 can be configured to perform feature extraction, generate the attention heat map and the complementary attention heat map, and perform the lesion identification, etc. using the artificial neural network module 420. In some examples, the optimization module 433 can obtain the total loss function based on the identification result of the lesion identification (including the first identification result, the second identification result, and the third identification result) to optimize the artificial neural network module 420. In this case, the identification result of the lesion identification can be obtained using the attention mechanism and the complementary attention mechanism, and the total loss function can be obtained based on the identification result of the lesion identification, so that the artificial neural network module 420 can be optimized using the total loss function, thereby improving the accuracy of the lesion identification of the artificial neural network module 420.

[0156] In some examples, the training method of the artificial neural network module 420 can be weakly supervised. In this case, the artificial neural network module 420 can be used to obtain a large amount of information from a small amount of information. In some examples, when the annotation result is a text annotation, the identification result can include the position and size of the lesion area. In some examples, the training method of the artificial neural network module 420 can also be unsupervised, semi-supervised, reinforcement learning, etc.

[0157] In some examples, the artificial neural network module 420 can be trained using the first loss function, the second loss function, and the third loss function. It should be noted that since the training model and the loss function involved are generally complex, the model generally has no analytical solution, and in some examples, the model parameters can be iterated a limited number of times to reduce the value of the loss function as much as possible, i.e., to obtain the analytical solution of the model, by using an optimization algorithm such as batch gradient descent (BGD), stochastic gradient descent (SGD), etc. In some examples, the artificial neural network module 420 can be trained using a backpropagation algorithm, in which case the network parameters can be obtained to minimize the error, thereby improving the recognition accuracy.

[0158] Figure 8 FIG. 4 is a flowchart illustrating a training method of an artificial neural network-based lesion identification according to an example of the present disclosure.

[0159] In some examples, as shown in FIG. 4, the training system 430 can be used to train the artificial neural network module 420. In some examples, the training system 430 can be configured to obtain a training dataset, and the training dataset can be used to train the artificial neural network module 420. In some examples, the training dataset can include a plurality of training samples, and each training sample can include a plurality of training images and a plurality of annotation results corresponding to the plurality of training images. In some examples, the plurality of training images can be used as input of the artificial neural network module 420, and the plurality of annotation results can be used as a label of the artificial neural network module 420. Figure 8As shown, the training method can include preparing a training data set (step S100); inputting the training data set into the artificial neural network module 420 and obtaining the first identification result, the second identification result and the third identification result matched with each examination image (step S200); calculating the total loss function based on the first identification result, the second identification result and the third identification result (step S300) and optimizing the artificial neural network module 420 using the total loss function (step S400). In this case, the first identification result, the second identification result and the third identification result can be obtained, and the total loss function can be obtained based on the first identification result, the second identification result and the third identification result, so that the artificial neural network module 420 can be optimized using the total loss function, and the accuracy of the organization lesion identification of the artificial neural network module 420 can be improved.

[0160] In step S100, the training data set can be prepared. In some examples, the training data set can include a plurality of examination images and a lesion-annotated result or a non-lesion-annotated result associated with the examination images.

[0161] In some examples, the training data set can include a plurality of examination images and an annotated image associated with the examination images.

[0162] In some examples, the examination images can be 500,000-2,000,000 tissue images from cooperating hospitals and removing patient information. In some examples, the examination images can be tissue images taken from CT scans, PET-CT scans, SPECT scans, MRIs, ultrasounds, X-rays, mammograms, angiograms, fluorograms, capsule endoscopy or a combination thereof. In some examples, the examination images can be fundus images. In some examples, the examination images can be composed of lesion regions and non-lesion regions. In some examples, the examination images can be used for training of the artificial neural network module 420.

[0163] In some examples, the examination images can be obtained by the obtaining module 410.

[0164] In some examples, the annotated images can include a lesion-annotated result or a non-lesion-annotated result. In some examples, the annotated result can be used as a true value to measure the size of the loss function.

[0165] In some examples, the annotated result can be image annotation or text annotation. In some examples, the image annotation can be an annotation box manually annotated to frame the lesion region.

[0166] In some examples, the annotation box can be a fixed shape, such as a triangle, a circle or a quadrilateral, etc. In some examples, the annotation box can also be an irregular shape based on the outline of the lesion region.

[0167] In some examples, the text label can be a determination result of whether the image has a lesion. For example, "with lesion" or "without lesion". In some examples, the text label can also be a type of lesion. For example, in the case of checking the image is a fundus image, the text label can be "maculopathy", "hypertensive retinopathy" or "diabetic retinopathy", etc.

[0168] In some examples, the training data set can be stored in the storage module 431. In some examples, the storage module 431 can be configured to store the training data set.

[0169] In some examples, the training data set can include 30%-60% of the inspection images with no lesion result label. In some examples, the training data set can include 10%, 20%, 30%, 40%, 50% or 60% of the inspection images with no lesion result label.

[0170] In some examples, the training data set can be stored using the storage module 431. In some examples, the storage module 431 can include the memory 20.

[0171] In some examples, the storage module 431 can be configured to store the inspection images and the labeled images associated with the inspection images.

[0172] In some examples, the artificial neural network module 420 can receive the training data set stored by the storage module 431.

[0173] In some examples, the training data set can be preprocessed.

[0174] In step S200, the training data set can be input into the artificial neural network module 420, and the first identification result, the second identification result and the third identification result matched with each inspection image can be obtained. In some examples, the training data set can be input into the artificial neural network module 420 to obtain the feature map, the attention heat map and the complementary attention heat map. In some examples, the feature extraction can be performed on the inspection image to obtain the feature map. In some examples, the feature map can be processed based on the attention mechanism to obtain the attention heat map. In some examples, the attention heat map can be processed based on the complementary attention mechanism to obtain the complementary attention heat map.

[0175] In some examples, step S200 can be implemented using the processing module 432. In some examples, the processing module 432 can include at least one processor 10.

[0176] In some examples, as described above, the artificial neural network module 420 can include the first artificial neural network 421, the second artificial neural network 422 and the third artificial neural network 423.

[0177] In some examples, the processing module 432 can be configured to perform feature extraction on the inspection image to obtain a feature map using the first artificial neural network 421. In some examples, the processing module 432 can be configured to obtain an attention heat map indicating the lesion region and a complementary attention heat map indicating the non-lesion region using the second artificial neural network 422.

[0178] In some examples, the processing module 432 can be configured to obtain a recognition result including tissue lesion recognition using the third artificial neural network 423. As described above, the third artificial neural network 423 can include an output layer. In some examples, the output layer can be configured to output a recognition result reflecting the inspection image. In this case, the third artificial neural network 423 can output a recognition result reflecting the inspection image.

[0179] In some examples, the processing module 432 can perform recognition on the inspection image based on the feature map using the third artificial neural network 423 to obtain a first recognition result.

[0180] In some examples, the processing module 432 can perform recognition on the inspection image based on the feature map and the attention heat map using the third artificial neural network 423 to obtain a second recognition result.

[0181] In some examples, the processing module 432 can perform recognition on the inspection image based on the feature map and the complementary attention heat map using the third artificial neural network 423 to obtain a third recognition result.

[0182] In some examples, the tissue lesion can be a fundus lesion. In this case, the artificial neural network module 420 can be used for fundus lesion recognition of a fundus image.

[0183] In step S300, a total loss function can be calculated based on the first recognition result, the second recognition result, and the third recognition result.

[0184] In some examples, step S300 can be implemented using the optimization module 433.

[0185] In some examples, the optimization module 433 can obtain a total loss function of the artificial neural network module 420 based on the first loss function, the second loss function, and the third loss function. In this case, the artificial neural network module 420 can be optimized using the total loss function.

[0186] In some examples, the optimization module 433 can combine the first recognition result with the labeled image to obtain a first loss function when the attention mechanism is not used. In some examples, the first loss function can be used to evaluate the degree of inconsistency between the recognition result of the examination image and the labeled result when the attention mechanism is not used. In this case, the accuracy of the recognition of the tissue lesion by the artificial neural network module 420 when the attention mechanism is not used can be improved.

[0187] In some examples, the optimization module 433 can combine the second recognition result with the labeled image to obtain a second loss function when the attention mechanism is used. In some examples, the second loss function can be used to evaluate the degree of inconsistency between the recognition result of the examination image and the labeled result when the attention mechanism is used. In this case, the accuracy of the recognition of the tissue lesion by the artificial neural network module 420 when the attention mechanism is used can be improved.

[0188] In some examples, the optimization module 433 can combine the third recognition result with the labeled image with the labeled result of no lesion to obtain a third loss function when the complementary attention mechanism is used. In some examples, the third loss function can be used to evaluate the degree of inconsistency between the recognition result of the examination image and the no-lesion recognition when the complementary attention mechanism is used. In this case, the accuracy of the recognition of the tissue lesion by the artificial neural network module 420 when the complementary attention mechanism is used can be improved.

[0189] In some examples, the first loss function, the second loss function, and the third loss function can be obtained by an error loss function. In some examples, the error loss function can be a function for evaluating the relevance between the true value (i.e., the labeled result) and the predicted value (i.e., the recognition result), such as a correlation function, an L1 loss function, an L2 loss function, or a Huber loss function.

[0190] In some examples, the total loss function can include a first loss term, a second loss term, and a third loss term.

[0191] In some examples, the first loss term can be positively correlated with the first loss function. In this case, the degree of inconsistency between the recognition result of the examination image and the labeled result when the attention mechanism is not used can be evaluated by using the first loss term, so that the accuracy of the recognition of the tissue lesion can be improved.

[0192] In some examples, the second loss term can be positively correlated with the difference between the second loss function and the first loss function. In some examples, when the second loss function is smaller than the first loss function, the second loss term can be a constant value. In this case, the degree of inconsistency between the recognition result of the examination image when the attention mechanism is used and the recognition result when the attention mechanism is not used can be evaluated by using the second loss term.

[0193] In some examples, the second loss term can be positively related to the difference between the second loss function and the first loss function. Specifically, when the difference between the second loss function and the first loss function is greater than zero, the difference between the second loss function and the first loss function can be taken as the second loss term, and when the difference between the second loss function and the first loss function is less than zero, the second loss term can be set to zero. In this case, the degree of inconsistency between the first recognition result and the second recognition result can be evaluated by using the second loss term, so that the second recognition result can be made closer to the labeled result relative to the first recognition result.

[0194] In some examples, the third loss term can be positively related to the third loss function. In this case, the degree of inconsistency between the third recognition result of the examination image when using the complementary attention mechanism and the labeled result without lesions can be evaluated by using the third loss term, so that the occurrence of misjudgment or omission can be reduced.

[0195] In some examples, the total loss function can further include a fourth loss term. In some examples, the fourth loss term can be a regularization term. In some examples, the fourth loss term can be a regularization term for the attention heat map. In some examples, the regularization term can be obtained based on total variation. In this case, overfitting of the artificial neural network module 420 can be inhibited.

[0196] In some examples, the total loss function can include loss term weight coefficients matched with respective loss terms. In some examples, the total loss function can further include a first loss term weight coefficient matched with the first loss term, a second loss term weight coefficient matched with the second loss term, a third loss term weight coefficient matched with the third loss term, and a fourth loss term weight coefficient matched with the fourth loss term, etc.

[0197] In some examples, the first loss term can be multiplied by the first loss term weight coefficient, the second loss term can be multiplied by the second loss term weight coefficient, the third loss term can be multiplied by the third loss term weight coefficient, the fourth loss term can be multiplied by the fourth loss term weight coefficient, and the fifth loss term can be multiplied by the fifth loss term weight coefficient. In this way, the influence degree of each loss term on the total loss function can be adjusted by the loss term weight coefficient.

[0198] In some examples, the loss term weight coefficient can be set to 0. In some examples, the loss term weight coefficient can be set to a positive number. In this case, since each loss term is a non-negative number, the value of the total loss function can be no less than zero.

[0199] In some examples, the function formula of the total loss function can be:

[0200]

[0201] wherein, L is a total loss function, λ1 is a first loss term weight coefficient, λ2 is a second loss term weight coefficient, λ3 is a third loss term weight coefficient, λ4 is a fourth loss term weight coefficient, f is an error loss function, X is an examination image, F(X) is a feature map generated after the examination image X passes through the first artificial neural network 421, l(X) is a label result of the examination image X, max is a maximum value function, C is a classifier function for outputting a recognition result based on an input feature map or a feature combination set, margin is a preset parameter, l0 is a label result of no lesion, M(X) is an attention heat map matched with the examination image X, is a complementary attention heat map matched with the examination image X, “·” in the function formula of the total loss function is a dot product operation of a matrix, and Regularize(M) is a regularization term for the attention heat map M. In some examples, the classifier function can be implemented by the third artificial neural network 423.

[0202] In some examples, the optimization module 433 can obtain a total loss function including a first loss term based on the first loss function, a second loss term based on a difference between the second loss function and the first loss function, and a third loss term based on the third loss function by using the first loss function, the second loss function, and the third loss function, and optimize the artificial neural network module 420 by using the total loss function.

[0203] In some examples, the total loss function can further include a fifth loss term. In some examples, the fifth loss term can be a total area term of the attention heat map. Specifically, the total area term of the attention heat map can be an area of a region determined as a lesion region in the attention heat map. In some examples, the total area term of the attention heat map M(X) can be represented by a formula SUM(M(X)). In some examples, the artificial neural network module 420 can be trained by using the fourth loss term to make the lesion region in the attention heat map smaller. In this case, the fifth loss term can be used to evaluate the area of the lesion region in the attention heat map and control the number of pixels in the attention heat map that have a greater impact on the recognition result, so as to limit the attention of the network on the pixels that have a greater impact on the recognition result. Thus, the accuracy of lesion region recognition can be increased.

[0204] In some examples, the total loss function can further include a sixth loss term. In some examples, the sixth loss term can be used to evaluate the degree of inconsistency between a bounding region of a lesion region in the recognition result and a label bounding box of a lesion region manually labeled in a label image.

[0205] In step S400, the artificial neural network module 420 can be optimized by using the total loss function.

[0206] In some examples, step S400 can be implemented by the optimization module 433.

[0207] In some examples, the optimization module 433 can optimize the artificial neural network module 420 by using the total loss function to minimize the total loss function. In this case, the total loss function can be minimized to improve the accuracy of the artificial neural network module 420 in identifying the tissue lesion.

[0208] In some examples, the optimization module 433 can obtain the total loss function based on the first loss term, the second loss term, the third loss term, and the total area term of the attention heat map, and optimize the artificial neural network module 420 by using the total loss function to obtain the artificial neural network module 420 that can be used for tissue lesion identification. In this way, the accuracy of the artificial neural network module 420 in identifying the tissue lesion can be further improved.

[0209] In some examples, the optimization module 433 can adjust the total loss function by changing the weights of the first loss term, the second loss term, the third loss term, and the fourth loss term.

[0210] In some examples, the optimization module 433 can use the first loss term and the sixth loss term as the total loss function (i.e., set the loss term weight coefficients of other loss terms to zero) to optimize the artificial neural network module 420. In this way, the accuracy of the attention heat map and the complementary attention heat map generated by the second artificial neural network 422 can be improved.

[0211] In some examples, during the optimization process, the loss term weight coefficients in the total loss function can be modified.

[0212] In some examples, the optimization module 433 can use an optimization algorithm to perform multiple iterations on the parameters in the total loss function to reduce the value of the total loss function. For example, in the present embodiment, a mini-batch stochastic gradient descent algorithm can be used to randomly select a group of input function parameters, and then perform multiple iterations on the parameters to reduce the value of the loss function.

[0213] In some examples, the training is paused when the total loss function is less than a second preset value or the number of iterations exceeds a third preset value.

[0214] In some examples, the optimization module 433 can first pre-train the artificial neural network module 420 without using the attention mechanism, and then train the artificial neural network module 420 using the attention mechanism. In this case, the training speed can be accelerated.

[0215] In some examples, the optimization module 433 can train the first artificial neural network 421, the second artificial neural network 422, and the third artificial neural network 423 simultaneously. In this case, the training speed can be accelerated.

[0216] In some examples, after the training is completed, the optimization module 433 can employ, for example, 0-20,000 tissue images (e.g., fundus images) as test tissue images to constitute a test set.

[0217] In some examples, the test tissue images can be used for post-training testing of the artificial neural network module 420.

[0218] FIG. 9(a) is a schematic diagram showing an example of a lesion area of a fundus image trained without using an attention mechanism according to an example of the present disclosure. FIG. 9(b) is a schematic diagram showing an example of a lesion area of a fundus image trained using a complementary attention mechanism according to an example of the present disclosure.

[0219] In some examples, the accuracy of tissue lesion recognition of a fundus image trained using a complementary attention mechanism is higher. As an example of training without using an attention mechanism, FIG. 9(a) shows a lesion area A of a fundus image trained without using an attention mechanism. As an example of training using a complementary attention mechanism, FIG. 9(b) shows a lesion area B of a fundus image trained using a complementary attention mechanism.

[0220] Although the present disclosure is specifically described above in connection with the drawings and examples, it will be understood that the above description is not intended to limit the present disclosure in any form. Those skilled in the art can make modifications and changes to the present disclosure as needed without departing from the spirit and scope of the present disclosure, and such modifications and changes fall within the scope of the present disclosure.

Claims

1. A method for identifying tissue lesions based on complementary attention mechanisms, characterized in that, include: Acquire tissue images; An artificial neural network module is used to receive the tissue image and identify lesions in the tissue image. The artificial neural network module includes a first artificial neural network, a second artificial neural network, and a third artificial neural network. The first artificial neural network is configured to extract features from the tissue image to obtain a feature map. The second artificial neural network is configured to obtain an attention heatmap indicating lesion regions. The third artificial neural network is configured to identify the tissue image based on the feature map. The training steps of the artificial neural network module include: preparing a training dataset, which includes multiple examination images and labeled images associated with the examination images. The labeled images include labels indicating lesions or labels indicating no lesions. The first artificial neural network is used to extract features from the examination images to obtain a feature map. The image is constructed by using the second artificial neural network to obtain an attention heatmap indicating lesion areas and a complementary attention heatmap indicating non-lesion areas. Pixels at the same position in the attention heatmap and complementary attention heatmap are inversely correlated. The examination image consists of the lesion areas and the non-lesion areas. The third artificial neural network is used to identify the examination image based on the feature map and the complementary attention heatmap to obtain a third identification result. The third identification result is combined with the labeled image with a lesion-free label to obtain a third loss function when using the complementary attention mechanism. The third loss function is used to evaluate the degree of inconsistency between the identification result of the examination image using the complementary attention mechanism and the lesion-free label, and the artificial neural network module is optimized using the third loss function.

2. The identification method as described in claim 1, characterized in that, It also includes preprocessing the tissue image and inputting the preprocessed tissue image into the artificial neural network module.

3. The identification method as described in claim 1, characterized in that, The artificial neural network module includes a backbone neural network and a second artificial neural network. The backbone neural network includes a first artificial neural network, a third artificial neural network, and a feature combination module.

4. The identification method as described in claim 3, characterized in that, The third artificial neural network outputs a feature combination set using the feature map, the attention heatmap, and the complementary attention heatmap. The third artificial neural network receives the feature combination set and outputs the identification result of tissue lesion recognition in the tissue image.

5. The identification method as described in claim 1, characterized in that, The first artificial neural network includes an input layer, an intermediate layer, and an output layer connected in sequence. The input layer is configured to receive a preprocessed tissue image, the intermediate layer is configured to extract the feature map based on the tissue image, and the output layer is configured to output the feature map.

6. The identification method as described in claim 1, characterized in that, The second artificial neural network includes an input layer, an intermediate layer, and an output layer connected in sequence. The input layer is configured to receive the recognition result obtained by the feature map and a portion of the weights of the third artificial neural network or the tissue lesion recognition result. The intermediate layer is configured to obtain feature weights based on the portion of the weights of the third artificial neural network or the tissue lesion recognition result. The intermediate layer is configured to generate the attention heatmap and / or the complementary attention heatmap based on the feature map and the feature weights. The output layer is configured to output the attention heatmap and / or the complementary attention heatmap.

7. The identification method as described in claim 1, characterized in that, The third artificial neural network includes an input layer, an intermediate layer, and an output layer connected in sequence, wherein the output layer is configured to output a recognition result reflecting the inspection image.

8. The identification method as described in claim 1, characterized in that, The artificial neural network module is trained using weak supervision.

9. The identification method as described in claim 1, characterized in that, The tissue lesion is a fundus lesion.

Images