Optimization Method of Artificial Neural Network Module

Abstract

The present disclosure describes an optimization method for an artificial neural network module, which is used for fundus lesion recognition. The optimization method includes: preparing a training data set, the training data set includes multiple inspection images with lesions and annotated images with lesion annotation results associated with the inspection images; extracting features from the inspection images to obtain feature maps, and processing the inspection images based on an attention mechanism to obtain an attention heat map; using a first artificial neural network to classify the inspection images, and combining the annotated images to obtain a first loss function, using a second artificial neural network module to classify the inspection images based on the feature map and the attention heat map, and combining the annotation results to obtain a second loss function, and using a third artificial neural network to discriminate the inspection images from being disease-free to obtain a third loss function. By combining these three loss functions, the accuracy of tissue lesion recognition can be effectively improved.

Description

[0001] This application is a divisional application with application date of November 27, 2020, application number 2020113599702, and invention name as Training method and training system for tissue lesion identification based on artificial neural network. Technical Field

[0002] The present disclosure generally relates to methods for optimizing artificial neural network modules. Background Art

[0003] With the development and maturity of artificial intelligence technology, artificial intelligence technology has gradually been promoted in various aspects of the medical field. In particular, medical imaging in medicine is currently a popular field for the application of artificial intelligence technology. Medical imaging is a useful tool for diagnosing many diseases. A large amount of medical image data is generated during the medical imaging process. It takes a lot of time for doctors to process and identify these image data, and it is difficult to ensure the accuracy of recognition. In medical images, artificial intelligence technology is mainly used to identify tissue lesions in the image to improve the accuracy of tissue lesion recognition.

[0004] At present, convolutional neural networks (CNNs) are commonly used in the application of artificial intelligence technology to identify medical images. The convolution structure of convolutional neural networks can reduce the amount of memory occupied by deep networks. It has three key operations: local receptive field, weight sharing, and pooling layer. In this way, the number of network parameters can be effectively reduced, alleviating the overfitting problem of convolutional neural networks. The structure of convolutional neural networks can better adapt to the structure of medical images and extract and identify features.

[0005] However, for some lesion sites, such as fundus lesions, the lesion areas are relatively small and irregularly distributed. The general convolutional neural network using the attention mechanism tends to ignore the lesion areas with low attention in the attention heat map, resulting in misjudgment, thus making the accuracy of tissue lesion recognition in these lesion areas low. Summary of the invention

[0006] The present disclosure is made in view of the above-mentioned state of the prior art, and its purpose is to provide a training method and training system for tissue lesion recognition based on artificial neural network that can effectively improve the accuracy of tissue lesion recognition.

[0007] To this end, the first aspect of the present disclosure provides a training method for tissue lesion recognition based on an artificial neural network, characterized in that it includes: preparing a training data set, the training data set includes multiple inspection images and annotated images associated with the inspection images, the annotated images include annotated results with lesions or annotated results without lesions; inputting the training data set into an artificial neural network module to perform feature extraction on the inspection image to obtain a feature map, and processing the feature map based on an attention mechanism to obtain an attention heat map, and processing the attention heat map based on a complementary attention mechanism to obtain a complementary attention heat map; the artificial neural network module includes a first artificial neural network, a second artificial neural network and a third artificial neural network; using the first artificial neural network to perform feature extraction on the inspection image to obtain the feature map, using the second artificial neural network to obtain the attention heat map indicating the lesion area and the complementary attention heat map indicating the non-lesion area, the inspection image is composed of the lesion area and the non-lesion area, and using the third artificial neural network based on the feature map The inspection image is recognized to obtain a first recognition result, the inspection image is recognized by the third artificial neural network based on the feature map and the attention heat map to obtain a second recognition result, and the inspection image is recognized by the third artificial neural network based on the feature map and the complementary attention heat map to obtain a third recognition result; the first recognition result is combined with the annotated image to obtain a first loss function when the attention mechanism is not used, the second recognition result is combined with the annotated image to obtain a second loss function when the attention mechanism is used, the third recognition result is combined with the annotated image with a lesion-free annotation result to obtain a third loss function when the complementary attention mechanism is used, the first loss function, the second loss function, and the third loss function are used to obtain a total loss function including a first loss term based on the first loss function, a second loss term based on the difference between the second loss function and the first loss function, and a third loss term based on the third loss function, and the total loss function is used to optimize the artificial neural network module. In this case, the first recognition result, the second recognition result and the third recognition result can be obtained, and the total loss function can be obtained based on the first recognition result, the second recognition result and the third recognition result, so that the total loss function can be used to optimize the artificial neural network module, thereby improving the accuracy of tissue lesion recognition of the artificial neural network module.

[0008] In addition, in the training method for tissue lesion recognition based on artificial neural network involved in the first aspect of the present disclosure, optionally, the total loss function also includes a total area term of the attention heat map, and the total area term is used to evaluate the area of ​​the lesion area. In this case, the fifth loss term can be used to evaluate the area of ​​the lesion area in the attention heat map and control the number of pixels in the attention heat map that have a greater impact on the recognition result, so that the network's attention is limited to the pixels that have a greater impact on the recognition result.

[0009] In addition, in the training method for tissue lesion recognition based on artificial neural network involved in the first aspect of the present disclosure, optionally, the total loss function also includes a regularization term for the attention heat map. In this case, overfitting of the artificial neural network module can be suppressed.

[0010] In addition, in the training method for tissue lesion recognition based on artificial neural network involved in the first aspect of the present disclosure, optionally, the first artificial neural network, the second artificial neural network and the third artificial neural network are trained simultaneously. In this case, the training speed can be accelerated.

[0011] In addition, in the training method for tissue lesion recognition based on an artificial neural network involved in the first aspect of the present disclosure, optionally, the third artificial neural network includes an input layer, an intermediate layer, and an output layer connected in sequence, and the output layer is configured to output a recognition result reflecting the inspection image. In this case, the third artificial neural network can be used to output a recognition result reflecting the tissue image.

[0012] In addition, in the training method for tissue lesion recognition based on artificial neural network involved in the first aspect of the present disclosure, optionally, the training mode of the artificial neural network module is weak supervision. In this case, the recognition result with more information can be obtained through the artificial neural network module using the annotation result with less information.

[0013] In addition, in the training method for tissue lesion recognition based on an artificial neural network involved in the first aspect of the present disclosure, optionally, the first loss function is used to evaluate the degree of inconsistency between the recognition result of the inspection image when the attention mechanism is not used and the annotation result. In this case, the accuracy of tissue lesion recognition of the artificial neural network module when the attention mechanism is not used can be improved.

[0014] In addition, in the training method for tissue lesion recognition based on an artificial neural network involved in the first aspect of the present disclosure, optionally, the second loss function is used to evaluate the degree of inconsistency between the recognition result of the inspection image when using the attention mechanism and the annotation result. In this case, the accuracy of tissue lesion recognition of the artificial neural network module when using the attention mechanism can be improved.

[0015] In addition, in the training method for tissue lesion recognition based on an artificial neural network involved in the first aspect of the present disclosure, optionally, the third loss function is used to evaluate the degree of inconsistency between the recognition result of the inspection image when using the complementary attention mechanism and the annotation result without lesions. In this case, the accuracy of tissue lesion recognition of the artificial neural network module when using the complementary attention mechanism can be improved.

[0016] In addition, in the training method for tissue lesion recognition based on artificial neural network involved in the first aspect of the present disclosure, optionally, the total loss function is used to optimize the artificial neural network module to minimize the total loss function. In this case, the total loss function can be minimized to improve the accuracy of tissue lesion recognition of the artificial neural network module.

[0017] In addition, in the training method for tissue lesion recognition based on artificial neural network involved in the first aspect of the present disclosure, optionally, the tissue lesion is a fundus lesion. In this case, the recognition result of the fundus lesion in the fundus image can be obtained by using the artificial neural network module.

[0018] The second aspect of the present disclosure provides a training system for tissue lesion recognition based on artificial neural network, characterized in that the training method provided by the first aspect of the present disclosure is used for training. In this case, the training system can be used to train the artificial neural network module.

[0019] According to the present disclosure, a training method and a training system for tissue lesion recognition based on an artificial neural network can be provided, which can effectively improve the accuracy of identifying tissue lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Embodiments of the present disclosure will now be explained in further detail, by way of example only, with reference to the accompanying drawings, in which:

[0021] Figure 1 Schematic diagram of an electronic device according to an example of the present disclosure.

[0022] Figure 2 FIG. 1 is a diagram showing an organizational image involved in the example of the present disclosure.

[0023] Figure 3is a structural block diagram showing an identification system for tissue lesion identification based on artificial neural network involved in the example of the present disclosure.

[0024] Figure 4 is a block diagram showing an example of an artificial neural network module involved in the examples of the present disclosure.

[0025] Figure 5 is a block diagram showing a modified example of the artificial neural network module involved in the example of the present disclosure.

[0026] Figure 6 is a schematic diagram showing the structure of the first artificial neural network involved in the example of the present disclosure.

[0027] Figure 7 is a structural block diagram showing a training system for tissue lesion recognition based on artificial neural network involved in the example of the present disclosure.

[0028] Figure 8 is a flow chart showing a training method for tissue lesion recognition based on an artificial neural network involved in an example of the present disclosure.

[0029] Figure 9(a) is a schematic diagram showing an example of a fundus image obtained without using the attention mechanism training involved in the example of the present disclosure.

[0030] Figure 9(b) is a schematic diagram showing an example of a lesion area of ​​a fundus image obtained by training using the complementary attention mechanism involved in the example of the present disclosure.

[0031] Main reference numerals: 1…electronic device, 10…processor, 20…memory, 30…computer program, 40…recognition system, 410…acquisition module, 4200…backbone neural network, 420…artificial neural network module, 421…first artificial neural network, 422…second artificial neural network, 423…third artificial neural network, 424…feature combination module, 430…training system, 431…storage module, 432…processing module, 433…optimization module, C1…first convolution layer, C2…second convolution layer, C3…third convolution layer, S1…first pooling layer, S2…second pooling layer, S3…third pooling layer DETAILED DESCRIPTION

[0032] Hereinafter, the preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same symbols are assigned to the same components, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the dimensions of the components or the shapes of the components may be different from the actual ones.

[0033] Figure 1 Schematic diagram of an electronic device according to an embodiment of the present disclosure.

[0034] like Figure 1 As shown, the recognition system 40 for tissue lesion recognition based on artificial neural network involved in the present disclosure can be carried by an electronic device 1 (such as a computer). In some examples, the electronic device 1 may include one or more processors 10, a memory 20, and a computer program 30 arranged in the memory 20. Among them, the one or more processors 10 may include a central processing unit, a graphics processing unit, and any other electronic components capable of processing data. For example, the processor 10 can execute instructions stored on the memory 20.

[0035] In some examples, the memory 20 may be a computer-readable medium that can be used to carry or store data. In some examples, the memory 20 may include, but is not limited to, a non-volatile memory or a flash memory. In some examples, the memory 20 may also be, for example, a ferroelectric random access memory (FeRAM), a magnetic random access memory (MRAM), a phase change random access memory (PRAM), or a resistive random access memory (RRAM). Thus, the possibility of data loss due to sudden power failure can be reduced.

[0036] In other examples, the memory 20 may also be other types of readable storage media, such as read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electronically erasable programmable read-only memory (EEPROM), and compact disc read-only memory (CD-ROM).

[0037] In some examples, the memory 20 may be an optical disk memory, a magnetic disk memory or a magnetic tape memory, so that a suitable memory 20 can be selected according to different situations.

[0038] In some examples, the computer program 30 may include instructions executed by one or more processors 10, and the instructions may be executed to enable the recognition system 40 to perform tissue lesion recognition on the tissue image. In some examples, the computer program 30 may be deployed in a local computer or in a cloud server.

[0039] In some examples, the computer program 30 may be stored in a computer-readable medium. The computer-readable storage medium may include one or more of a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, or a magnetic storage device.

[0040] Figure 2 FIG. 1 is a diagram showing an organizational image involved in the example of the present disclosure. Figure 3 is a block diagram showing the structure of an identification system 40 for tissue lesion identification based on an artificial neural network according to an example of the present disclosure.

[0041] In some examples, the recognition system 40 for tissue lesion recognition based on artificial neural network can be used to implement tissue lesion recognition of tissue images and obtain recognition results. In some examples, the recognition system 40 for tissue lesion recognition can also be referred to as recognition system 40.

[0042] In some examples, such as Figure 3 As shown, the recognition system 40 may include an acquisition module 410, an artificial neural network module 420, and a training system 430 for tissue lesion recognition based on an artificial neural network. In some examples, the acquisition module 410 may be used to acquire a tissue image. In some examples, the artificial neural network module 420 may be used to perform feature extraction, tissue lesion recognition, and other processing on the tissue image, and obtain a recognition result of tissue lesion recognition. In some examples, the training system 430 for tissue lesion recognition based on an artificial neural network may be used to train the artificial neural network module 420. In some examples, the training system 430 may utilize the first recognition result, the second recognition result, and the third recognition result obtained by the artificial neural network module 420, and obtain a total loss function based on the first recognition result, the second recognition result, and the third recognition result to optimize the artificial neural network module 420. In this case, the first recognition result, the second recognition result, and the third recognition result can be obtained, and the total loss function can be obtained based on the first recognition result, the second recognition result, and the third recognition result, so that the total loss function can be used to optimize the artificial neural network module 420, thereby improving the accuracy of tissue lesion recognition of the artificial neural network module 420.

[0043] In some examples, training system 430 for artificial neural network-based tissue lesion recognition may also be referred to as training system 430 .

[0044] In some examples, the recognition system 40 may further include a pre-processing module and a judgment module (not shown).

[0045] In some examples, the tissue image can be an image of a tissue cavity taken from a CT scan, a PET-CT scan, a SPECT scan, an MRI, an ultrasound, an X-ray, a mammogram, an angiogram, a fluorograph, an image of a capsule endoscopy, or a combination thereof. In some examples, the tissue image can be acquired by acquisition module 410.

[0046] In some examples, the acquisition module 410 may be configured to acquire a tissue image, which may be a tissue image acquired by an acquisition device such as a camera, an ultrasound imager, or an X-ray scanner.

[0047] In some examples, the tissue image may be, for example, a fundus image, an esophagus image, a stomach image, a large intestine image, a colon image, or a small intestine image. Figure 2 As shown, the tissue image may be a fundus image. In this case, the fundus image can be used to identify fundus lesions through the identification system 40.

[0048] In some examples, tissue lesion recognition may be identifying tissue lesions in a tissue image to obtain a recognition result.

[0049] In some examples, when the tissue image is a fundus image, the tissue lesion may be a fundus lesion. In this case, the artificial neural network module 420 can be used to obtain the recognition result of the fundus lesion of the fundus image.

[0050] In some examples, the tissue image may be composed of diseased areas and non-lesioned areas.

[0051] In some examples, tissue images (color images) with tissue lesions generally contain obvious features such as erythema and swelling. Therefore, trained artificial neural networks can be used to automatically extract and identify these features to help patients identify possible lesions. This can improve the accuracy and speed of identification, while reducing the problems of large errors and long time consumption caused by human doctors reading films one by one based on their own experience.

[0052] In some examples, when the tissue image is a fundus image, the tissue image can be classified according to function. For example, in the training step, the tissue image can be an inspection image or a labeling image (described later).

[0053] In some examples, the image input to the artificial neural network module 420 may be a tissue image. In this case, the artificial neural network module 420 may perform tissue lesion recognition on the tissue image.

[0054] In some examples, the recognition system 40 may be used to recognize tissue lesions in tissue images. In some examples, after the tissue images enter the recognition system 40, operations such as preprocessing, feature extraction, and tissue lesion recognition may be performed on the tissue images.

[0055] In some examples, the recognition system 40 may further include a preprocessing module and a judgment module. The preprocessing module may be used to preprocess the tissue image and input the preprocessed tissue image into the artificial neural network module 420.

[0056] In some examples, the preprocessing module can preprocess the tissue image. In some examples, the preprocessing can include at least one of region of interest detection, image cropping, resizing, and normalization. In this case, it is convenient for the subsequent artificial neural network module 420 to identify and judge tissue lesions on the tissue image. In some examples, the tissue image can be, for example, a fundus image, an esophageal image, a stomach image, a large intestine image, a colon image, or a small intestine image.

[0057] In some examples, the pre-processing module may include a region detection unit, an adjustment unit, and a normalization unit.

[0058] In some examples, the region detection unit can detect the region of interest from the tissue image. For example, if the tissue image is a fundus image, the fundus region centered on the optic disc, or the fundus region containing the optic disc and centered on the macula can be detected from the fundus image. In some examples, the region detection unit can detect the region of interest in the tissue image by, for example, a sampling threshold method or a Hough transform.

[0059] In some examples, the adjustment unit can be used to cut and resize the tissue image. Due to different devices or shooting conditions for collecting tissue images, the obtained tissue images may differ in resolution, size, etc. In this case, these tissue images can be cut and resized to reduce the differences. In some examples, the tissue image can be cut according to a specific shape. In some examples, the specific shape can include but is not limited to a square, rectangle, circle, or oval, etc.

[0060] In other examples, the size of the tissue image can be adjusted to a specified size by the adjustment unit. For example, the specified size can be 256×256, 512×512, or 1024×1024. However, the examples disclosed herein are not limited thereto. In other examples, the size of the tissue image can also be any other size. For example, the size of the tissue image can be 128×128, 768×768, or 2048×2048.

[0061] In some examples, the preprocessing module may include a normalization unit. The normalization unit may be used to perform normalization processing on multiple tissue images.

[0062] In some examples, the normalization method of the normalization unit is not particularly limited, for example, zero mean, unit standard deviation, etc. can be used. In addition, in some examples, normalization can also be performed within the range of [0,1]. In this case, the differences between different tissue images can be overcome through normalization.

[0063] In some examples, the normalization includes normalization of image format, image slice interval, image intensity, image contraction, and image orientation. In some examples, the tissue image can be normalized to a DICOM format, a NIfTI format, or a raw binary format.

[0064] Figure 4 is a block diagram showing an example of an artificial neural network module involved in the examples of the present disclosure.

[0065] As described above, the recognition system 40 may include an artificial neural network module 420. In some examples, the artificial neural network module 420 may be used to identify tissue lesions on tissue images. In some examples, the artificial neural network module 420 may include multiple artificial neural networks. In some examples, the artificial neural network may be trained using one or more processors 10. Generally speaking, an artificial neural network may include an artificial neuron or node, which may be used to receive a tissue image and perform operations on the tissue image based on weights, and then selectively pass the results of the operations to other neurons or nodes. Among them, weights may be associated with artificial neurons or nodes and simultaneously constrain the outputs of other artificial neurons. Weights (i.e., network parameters) may be determined by iteratively training an artificial neural network using a training data set (described later).

[0066] In some examples, such as Figure 4 As shown, the artificial neural network module 420 may include a trunk neural network 4200 and a second artificial neural network 422 .

[0067] In some examples, the backbone neural network 4200 may include a first artificial neural network 421 , a third artificial neural network 423 , and a feature combination module 424 .

[0068] In some examples, the first artificial neural network 421 may receive a tissue image and perform feature extraction on the tissue image to obtain a feature map.

[0069] In some examples, the second artificial neural network 422 can receive the feature map and the recognition result from the third artificial neural network 423 and obtain an attention heat map indicating the lesion area and a complementary attention heat map indicating the non-lesion area. It should be noted that in other examples, the above-mentioned attention heat map or complementary attention heat map can also be considered as a feature map.

[0070] In some examples, the feature combination module 424 may receive the feature map, the attention heat map, and the complementary attention heat map and output a feature combination set. In some examples, the feature combination module 424 may also directly output the feature map.

[0071] In some examples, the third artificial neural network 423 may receive a feature map or a feature combination set and output a recognition result of tissue lesion recognition of a tissue image.

[0072] In some examples, the tissue image (eg, a preprocessed tissue image) input to the artificial neural network module 420 may enter the first artificial neural network 421 , and ultimately the third artificial neural network 423 may output a recognition result.

[0073] Figure 5 is a block diagram showing a modified example of the artificial neural network module involved in the example of the present disclosure.

[0074] In addition, in some examples, such as Figure 5 As shown, the artificial neural network module 420 may include a trunk neural network 4200 and a second artificial neural network 422 .

[0075] In some examples, such as Figure 5 As shown, the backbone neural network 4200 may include a first artificial neural network 421 and a third artificial neural network 423 .

[0076] In some examples, the third artificial neural network 423 may have a feature combination function. For details, see the relevant description in the feature combination module 424.

[0077] In some examples, the first artificial neural network 421 may receive a tissue image and perform feature extraction on the tissue image to obtain a feature map.

[0078] In some examples, the second artificial neural network 422 may also obtain a complementary attention heat map indicating a non-lesion area based on the attention heat map.

[0079] In some examples, the third artificial neural network 423 can receive the feature map, the attention heat map, and the complementary attention heat map and output a recognition result of tissue lesion recognition of the tissue image. In some examples, the attention heat map can be a heat map indicating the lesion area obtained based on the attention mechanism. In some examples, the attention heat map can show the importance of each pixel in the tissue image when forming the feature map.

[0080] In some examples, the complementary attention heat map can be a heat map indicating non-lesion areas obtained based on the complementary attention mechanism.

[0081] In some examples, the complementary attention heatmap can be a complementary image of the attention heatmap. In some examples, the size and format of the complementary attention heatmap can be the same as the size and format of the attention heatmap.

[0082] As described above, the artificial neural network module 420 may include a first artificial neural network 421 (see Figure 5 ).

[0083] In some examples, first artificial neural network 421 may use one or more deep neural networks to automatically identify features in tissue images.

[0084] In some examples, the first artificial neural network 421 can be used to receive the tissue image preprocessed by the preprocessing module and generate one or more feature maps. In some examples, the first artificial neural network 421 can be, for example, by combining multiple layers of low-level features (pixel-level features). In this case, an abstract description of the tissue image can be achieved.

[0085] In some examples, the first artificial neural network 421 may include an input layer, an intermediate layer, and an output layer connected in sequence. The input layer may be configured to receive a tissue image preprocessed by a preprocessing module. The intermediate layer may be configured to extract a feature map based on the tissue image, and the output layer may be configured to output the feature map.

[0086] In some examples, the tissue image input to the artificial neural network module 420 can be converted into a pixel matrix, for example, a three-dimensional pixel matrix. The length and width of the three-dimensional matrix can represent the size of the image, and the depth of the three-dimensional matrix represents the color channel of the image. In some examples, the depth can be 1 (i.e., the tissue image is a grayscale image), and in some examples, the depth can be 3 (i.e., the tissue image is a color image in RGB color mode).

[0087] In some examples, the first artificial neural network 421 may use a convolutional neural network. Since the convolutional neural network has advantages such as local receptive field and weight sharing, it can greatly reduce the training of parameters, thereby improving the processing speed and saving hardware costs. In addition, the convolutional neural network can more effectively identify tissue images.

[0088] Figure 6 is a schematic diagram showing the structure of the first artificial neural network 421 involved in the example of the present disclosure.

[0089] In some examples, the first artificial neural network 421 may contain multiple intermediate layers, each of which may include multiple neurons or nodes, and each neuron or node in the intermediate layer may apply an activation function (such as a ReLU (rectified linear unit) function, a sigmoid function, or a tanh function, etc.) to the output of each neuron or node. The activation functions applied to different neurons affect the activation functions applied to other neurons.

[0090] In some examples, such as Figure 6 As shown, the middle layer of the first artificial neural network 421 may include multiple convolutional layers and multiple pooling layers. In some examples, the convolutional layers and the pooling layers may be combined alternately. In some examples, the tissue image may pass through the first convolutional layer C1, the first pooling layer S1, the second convolutional layer C2, the second pooling layer S2, the third convolutional layer C3, and the third pooling layer S3 in sequence. In this case, the tissue image can be subjected to convolution processing and pooling processing alternately.

[0091] In other examples, the first artificial neural network 421 may not include a pooling layer, thereby avoiding data loss during the pooling process and simplifying the network structure.

[0092] In some examples, the convolution layer can convolve the tissue image in the convolutional neural network using the convolution kernel. In this case, a more abstract feature can be obtained so that the matrix depth becomes deeper.

[0093] In some examples, the convolution kernel size may be 3*3. In other examples, the convolution kernel size may be 5*5. In some examples, a 5×5 convolution kernel may be used in the first convolution layer C1, and a 3×3 convolution kernel may be used in other convolution layers. In this case, the training efficiency can be improved. In some examples, the size of the convolution kernel may be set to any size. In this case, the size of the convolution kernel may be selected according to the size of the image and the computational cost.

[0094] In some examples, the pooling layer can also be called a downsampling layer. In some examples, the input tissue image can be processed using pooling methods such as max-pooling, mean-pooling, or stochastic-pooling. In this case, through the pooling operation, on the one hand, the feature dimension can be reduced and the operation efficiency can be improved. On the other hand, the convolutional neural network can also extract more abstract high-level features to improve the accuracy of tissue lesion recognition.

[0095] In addition, in some examples, in the above convolutional neural network, the number of convolutional layers and pooling layers can be increased accordingly according to the situation. In this case, the convolutional neural network can also extract more abstract high-level features to further improve the accuracy of tissue lesion recognition.

[0096] In some examples, after the preprocessed tissue image passes through the first artificial neural network 421, a feature map corresponding to the tissue image may be output. In some examples, the feature map may have multiple depths. In some examples, after the preprocessed tissue image passes through the first artificial neural network 421, multiple feature maps may be output. In some examples, the multiple feature maps may correspond to a feature respectively. In some examples, tissue lesions may be identified on the tissue image based on the features corresponding to the feature maps.

[0097] In some examples, before the first artificial neural network 421 outputs the feature map, the feature map can be sequentially deconvolved and upsampled. In some examples, the feature map can be subjected to multiple deconvolution and upsampling processes. For example, the feature map can sequentially pass through a first deconvolution layer, a first upsampling layer, a second deconvolution layer, a second upsampling layer, a third deconvolution layer, and a third upsampling layer. In this case, the size of the feature map can be changed, and the data information of part of the tissue image can be retained.

[0098] In some examples, the number of deconvolution layers can be the same as the number of convolution layers, and the number of pooling layers (downsampling layers) can be the same as the number of upsampling layers. Thus, the size of the feature map can be made the same as the tissue image.

[0099] In some examples, before the feature map passes through the deconvolution layer (upsampling), a tissue image processed by the convolution layer (pooling layer) can be selected for convolution. For example, before the feature map enters the second deconvolution layer (second upsampling layer), the feature map can be convolved with the output image of the second convolution layer C2 (second pooling layer S2). Before the feature map enters the third deconvolution layer (third upsampling layer), the feature map can be convolved with the output image of the first convolution layer C1 (first pooling layer S1). In this case, the data information lost when passing through the pooling layer or convolution layer can be supplemented.

[0100] In some examples, after a feature map is generated by the first artificial neural network 421 , an attention heat map matching the feature map may be generated by the second artificial neural network 422 .

[0101] In this embodiment, the second artificial neural network 422 is an artificial neural network with an attention mechanism. In some examples, the output image of the second artificial neural network 422 may include an attention heat map and a complementary attention heat map.

[0102] In some examples, the second artificial neural network 422 may include an input layer, an intermediate layer, and an output layer connected in sequence. The input layer is configured to receive the recognition result of the partial weights or tissue lesion recognition of the feature map and the third artificial neural network 423. The intermediate layer can be configured to obtain the feature weights based on the partial weights or tissue lesion recognition results of the third artificial neural network 423. The intermediate layer can be configured to generate an attention heat map and / or a complementary attention heat map based on the feature map and the feature weights. The output layer is configured to output an attention heat map and / or a complementary attention heat map. In some examples, the feature map can be generated by the first artificial neural network 421.

[0103] In some examples, the attention mechanism can selectively filter out a small amount of important information from a large amount of information in the input feature map and focus on this important information.

[0104] In some examples, the attention heat map may be an image that represents attention in the form of a heat map. Generally speaking, pixels at corresponding positions that are red or white in the attention heat map have a greater impact on the identification of tissue lesions in tissue images. Pixels at corresponding positions that are blue or black in the attention heat map have a smaller impact on the identification of tissue lesions in tissue images.

[0105] In some examples, feature weights may be used to weight each feature map and obtain an attention heat map. In some examples, feature weights may be obtained through an attention mechanism. In some examples, the attention mechanism may include, but is not limited to, a channel attention module (CAM), a gradient-based channel attention mechanism (Grad-CAM), a gradient-based enhanced channel attention mechanism (Grad-CAM++), or a spatial attention module (SAM).

[0106] In some examples, when the third artificial neural network 423 has a global pooling layer and a fully connected layer. In some examples, the feature weight can be the weight of the output layer of the third artificial neural network 423 through the fully connected layer to the third artificial neural network 423. For example, in the case where the tissue image is a fundus image, the third artificial neural network 423 can receive the feature map of the fundus image and obtain a first recognition result (described later). If the first recognition result is "macula", the weight of the recognition result of "macula" from each neuron or node in the global pooling layer in the fully connected layer is extracted as the feature weight.

[0107] In some examples, feature weights may be calculated based on the tissue lesion recognition result in the third artificial neural network 423. In some examples, the partial derivative of the first recognition result (e.g., the probability of tissue lesions) of the third artificial neural network 423 with respect to all pixels in a feature map may be calculated, and the partial derivatives of all pixels in the feature map may be globally pooled to obtain feature weights corresponding to the feature map.

[0108] In some examples, an attention heat map that matches the feature map can be generated by the second artificial neural network 422. In some examples, a complementary attention heat map can be generated by the second artificial neural network 422. In some examples, two pixel values ​​corresponding to pixels at the same position in the attention heat map and the complementary attention heat map are anti-correlated. In some examples, the attention heat map and / or the complementary attention heat map can be normalized. In some examples, the sum or product of two pixel values ​​corresponding to pixels at the same position in the attention heat map and the complementary attention heat map is a constant value.

[0109] In some examples, the attention heatmap and / or the complementary attention heatmap can be regularized using total variation.

[0110] In some examples, the feature combination module 424 may be connected to the output layers of the first artificial neural network 421 and the second artificial neural network 422 .

[0111] In some examples, the feature combination module 424 can have an input layer and an output layer. In some examples, the output layer of the feature combination module 424 can be a feature map or a feature combination set. In some examples, the input layer of the feature combination module 424 can receive a feature map, an attention heat map, or a complementary attention heat map.

[0112] In some examples, the feature combination module 424 may perform feature combination on the feature map output by the first artificial neural network 421 and the attention heat map or the complementary attention heat map output by the second artificial neural network 422 to form a feature combination set.

[0113] In some examples, the feature combination set may include at least one of a first feature combination set and a second feature combination set.

[0114] In some examples, the feature combination module 424 may perform feature combination on the feature map output by the first artificial neural network 421 and the attention heat map output by the second artificial neural network 422 to form a first feature combination set.

[0115] In some examples, the feature combination module 424 may perform feature combination on the feature map output by the first artificial neural network 421 and the complementary attention heat map output by the second artificial neural network 422 to form a second feature combination set.

[0116] In some examples, feature combination module 424 may directly output a feature map.

[0117] In some examples, the feature combination module 424 may also calculate the difference between the feature map and the attention heat map to obtain the first feature combination set.

[0118] In some examples, the feature combination module 424 can also calculate the difference between the feature map and the complementary attention heat map to obtain a second feature combination set:

[0119] In some examples, the feature combination module 424 may also calculate the convolution of the feature map and the attention heat map to obtain the first feature combination set.

[0120] In some examples, the feature combination module 424 may also calculate the convolution of the feature map and the complementary attention heat map to obtain a second feature combination set.

[0121] In some examples, the feature combination module 424 may also calculate the mean of the feature map and the attention heat map to obtain the first feature combination set.

[0122] In some examples, the feature combination module 424 may also calculate the mean of the feature map and the complementary attention heat map to obtain a second feature combination set.

[0123] In addition, in some other examples, the feature combination module 424 may perform a linear or nonlinear transformation on the feature map and the attention heat map to obtain a first feature combination set.

[0124] In addition, in some other examples, the feature combination module 424 may perform a linear or nonlinear transformation on the feature map and the complementary attention heat map to obtain a second feature combination set.

[0125] In some examples, the output layer of the feature combination module 424 can output a feature map, a first feature combination set, and a second feature combination set. In some examples, the feature map, the first feature combination set, and the second feature combination set output by the feature combination module 424 can be input into the third artificial neural network 423, and the third artificial neural network 423 performs tissue lesion recognition.

[0126] In some examples, the feature combination module 424 can be incorporated into the third artificial neural network 423 and be a part of the third artificial neural network 423. In this case, the artificial neural network module 420 can include the first artificial neural network 421, the second artificial neural network 422 and the third artificial neural network 423.

[0127] In some examples, where the feature combination module 424 is incorporated into the third artificial neural network 423, the input layer of the third artificial neural network 423 may receive a feature map, an attention heat map, or a complementary attention heat map.

[0128] In some examples, the third artificial neural network 423 may include an input layer, an intermediate layer, and an output layer connected in sequence. In some examples, the output layer is configured to be used to output a recognition result reflecting the tissue image. In this case, the third artificial neural network 423 can be used to output a recognition result reflecting the tissue image. In some examples, the output layer of the third artificial neural network 423 may include a Softmax layer. In some examples, the intermediate layer of the third artificial neural network 423 may be a fully connected layer.

[0129] In some examples, the final classification can be performed by the fully connected layer, and finally the probability of the tissue image belonging to each tissue lesion category is obtained through the Softmax layer. In this case, the recognition result of the tissue lesion recognition of the tissue image can be obtained based on the probability.

[0130] In some examples, the third artificial neural network 423 may include various linear classifiers, such as a single-layer fully connected layer.

[0131] In some examples, the third artificial neural network 423 may include various nonlinear classifiers, such as logistic regression, random forest, or support vector machines.

[0132] In some examples, the third artificial neural network 423 may include multiple classifiers. In some examples, the classifier may provide a recognition result of tissue lesion recognition of the tissue image. For example, when the tissue image is a fundus image, a recognition result of fundus lesion recognition of the fundus image may be provided. In this case, fundus lesion recognition can be performed on the fundus image.

[0133] In some examples, the output of the third neural network 423 may be a value between 0 and 1, which may be used to represent the probability that the tissue image belongs to the category of each tissue lesion.

[0134] In some examples, when the probability that a tissue image belongs to a certain tissue lesion category is the highest, the category is used as the recognition result of tissue lesion identification of the tissue image. For example, among the probabilities that a tissue image belongs to each tissue lesion category, if the probability that the category is no lesion is the highest, the recognition result of tissue lesion identification of the tissue image may be no lesion. For another example, in the process of identifying fundus lesions of fundus images, if the predicted probabilities of macula and no lesions output by the third artificial neural network 423 are 0.8 and 0.2 respectively, it can be considered that the fundus image has macular lesions.

[0135] In some examples, the third artificial neural network 423 can output a recognition result that matches the tissue image. In some examples, the recognition result can include a first recognition result when the attention mechanism is not used, a second recognition result when the attention mechanism is used, and a third recognition result when the attention mechanism and complementary attention are used.

[0136] In some examples, the third artificial neural network 423 can perform tissue lesion recognition on the feature map output by the feature combination module 424 and obtain a first recognition result.

[0137] In some examples, the third artificial neural network 423 may perform tissue lesion recognition on the first feature combination set output by the feature combination module 424 and obtain a second recognition result.

[0138] In some examples, the third artificial neural network 423 may perform tissue lesion recognition on the second feature combination set output by the feature combination module 424 and obtain a third recognition result.

[0139] In some examples, the recognition result may include two results: the presence of lesions and the absence of lesions. In some examples, the recognition result may also include the absence of lesions or a specific type of lesions. For example, in the case where the tissue image is a fundus image, the recognition result may include but is not limited to one of the following: absence of lesions, hypertensive retinopathy, or diabetic retinopathy. In this case, the recognition result of the fundus lesion recognition of the fundus image can be obtained. In some examples, the recognition results of a tissue image may be multiple. For example, the recognition results may be two results: hypertensive retinopathy and diabetic retinopathy.

[0140] In some examples, the recognition system 40 may further include a determination module.

[0141] In some examples, the judgment module may receive the output of the artificial neural network module 420. In this case, the judgment module may synthesize the output of the artificial neural network module 420 and output the final recognition result, thereby generating a summary report.

[0142] In some examples, the first recognition result can be used as the final recognition result of the tissue image. In this case, when the tissue image is recognized for tissue lesions using the artificial neural network module 420, the tissue image can be recognized for tissue lesions using the backbone neural network 4200 including the first artificial neural network 421 and the third artificial neural network 423, thereby accelerating the recognition speed.

[0143] In some examples, the second recognition result may be used as the final recognition result of the tissue image.

[0144] As described above, the third recognition result can be obtained based on the complementary attention mechanism. In some examples, the final recognition result of the tissue image can be obtained based on the first recognition result, the second recognition result, and the third recognition result. For example, in some examples, the final recognition result may include the second recognition result and the third recognition result. In some examples, the final recognition result may include the first recognition result and the third recognition result.

[0145] In some examples, the summary report generated by the judgment module may include at least one of the first recognition result, the second recognition result, the third recognition result, and the final recognition result. In some examples, the judgment module may color-code the tissue image based on the attention heat map to generate a lesion indication map to indicate the lesion area. The summary report generated by the judgment module may include the lesion indication map.

[0146] In some examples, the summary report generated by the judgment module may include the location of the corresponding lesion and mark the location with a marking box.

[0147] In some examples, the summary report generated by the judgment module can display the lesion area of ​​the tissue image in the form of a heat map. Specifically, in the heat map, the area with a high possibility of lesions can be red or white, and the area with a low possibility of lesions can be blue or black. In this case, the lesion area can be indicated in an intuitive manner.

[0148] In some examples, the judgment module can also be used to select the lesion area. In some examples, the lesion area can be selected by a fixed shape (such as a triangle, a circle, a quadrilateral, or other regular shapes). In some examples, the lesion area can also be outlined. In this case, the lesion area can be intuitively displayed.

[0149] In some examples, the judgment module can also be used to outline the lesion area. For example, the values ​​corresponding to each pixel in the attention heat map can be analyzed, and the pixel points with values ​​greater than a first preset value can be classified as the lesion area, and the pixel points with values ​​less than the first preset value can be classified as the non-lesion area.

[0150] The identification method of tissue lesion identification based on artificial neural network involved in the present disclosure is implemented by the identification system 40 .

[0151] In some examples, the recognition method includes: acquiring a tissue image and acquiring a recognition result of tissue lesion recognition using an artificial neural network module 420. In some examples, the tissue image may be a tissue image acquired by an acquisition device. In some examples, the artificial neural network module 420 is trained by a training system 430. In this case, the recognition result of tissue lesion recognition can be acquired using the artificial neural network module 420, and the artificial neural network module 420 can be optimized using a total loss function, thereby improving the accuracy of tissue lesion recognition.

[0152] The following describes in detail the training method (sometimes also referred to as training method) and training system for tissue lesion recognition based on artificial neural network involved in this embodiment in conjunction with the accompanying drawings.

[0153] In some examples, the training method can be implemented using a training system 430 for tissue lesion recognition based on an artificial neural network. In this case, the training system 430 can be used to train the artificial neural network module 420.

[0154] Figure 7 is a structural block diagram showing a training system 430 for tissue lesion recognition based on an artificial neural network involved in an example of the present disclosure.

[0155] In some examples, such as Figure 7As shown, the training system 430 may include a storage module 431, a processing module 432, and an optimization module 433. In some examples, the storage module 431 may be configured to store a training data set. In some examples, the processing module 432 may utilize the artificial neural network module 420 to perform operations such as feature extraction, generating an attention heat map and a complementary attention heat map, and tissue lesion recognition. In some examples, the optimization module 433 may obtain a total loss function based on the recognition results of tissue lesion recognition (including the first recognition result, the second recognition result, and the third recognition result) to optimize the artificial neural network module 420. In this case, the recognition results of tissue lesion recognition can be obtained using the attention mechanism and the complementary attention mechanism, and the total loss function can be obtained based on the recognition results of tissue lesion recognition, so that the total loss function can be used to optimize the artificial neural network module 420, thereby improving the accuracy of tissue lesion recognition of the artificial neural network module 420.

[0156] In some examples, the training method of the artificial neural network module 420 may be weak supervision. In this case, the labeling result with less information can be used to obtain the recognition result with more information through the artificial neural network module 420. In some examples, when the labeling result is a text labeling, the recognition result may include the location and size of the lesion area. In some examples, the training method of the artificial neural network module 420 may also be unsupervised, semi-supervised, reinforcement learning, etc.

[0157] In some examples, the artificial neural network module 420 can be trained using the first loss function, the second loss function, and the third loss function. It should be noted that since the training model and loss function involved are generally complex, the model generally has no analytical solution. In some examples, the value of the loss function can be reduced as much as possible by iterating the model parameters a finite number of times through an optimization algorithm (such as batch gradient descent (BGD), stochastic gradient descent (SGD), etc.), that is, the analytical solution of the model is obtained. In some examples, the artificial neural network module 420 can be trained using a back propagation algorithm. In this case, the network parameters with the minimum error can be achieved, thereby improving the recognition accuracy.

[0158] Figure 8 is a flow chart showing a training method for tissue lesion recognition based on an artificial neural network involved in an example of the present disclosure.

[0159] In some examples, such as Figure 8As shown, the training method may include preparing a training data set (step S100); inputting the training data set into the artificial neural network module 420, and obtaining a first recognition result, a second recognition result, and a third recognition result that match each inspection image (step S200); calculating a total loss function based on the first recognition result, the second recognition result, and the third recognition result (step S300), and optimizing the artificial neural network module 420 using the total loss function (step S400). In this case, the first recognition result, the second recognition result, and the third recognition result can be obtained, and the total loss function can be obtained based on the first recognition result, the second recognition result, and the third recognition result, so that the artificial neural network module 420 can be optimized using the total loss function, thereby improving the accuracy of tissue lesion recognition of the artificial neural network module 420.

[0160] In step S100, a training data set may be prepared. In some examples, the training data set may include a plurality of inspection images and annotation results of images with or without lesions associated with the inspection images.

[0161] In some examples, the training dataset may include a plurality of inspection images and annotated images associated with the inspection images.

[0162] In some examples, the inspection images may be 50,000 to 200,000 tissue images from a partner hospital with patient information removed. In some examples, the inspection images may be tissue images from CT scans, PET-CT scans, SPECT scans, MRI, ultrasound, X-rays, mammograms, angiograms, fluorograms, capsule endoscopy, or a combination thereof. In some examples, the inspection images may be fundus images. In some examples, the inspection images may be composed of lesion areas and non-lesion areas. In some examples, the inspection images may be used for training of the artificial neural network module 420.

[0163] In some examples, the inspection image may be acquired by acquisition module 410 .

[0164] In some examples, the labeled image may include a labeled result with a lesion or a labeled result without a lesion. In some examples, the labeled result may be used as a true value to measure the size of the loss function.

[0165] In some examples, the annotation result may be an image annotation or a text annotation. In some examples, the image annotation may be a manually annotated annotation box for selecting a lesion area.

[0166] In some examples, the annotation box may be a fixed shape, such as a regular shape such as a triangle, a circle, or a quadrilateral. In some examples, the annotation box may also be an irregular shape based on the outline of the lesion area.

[0167] In some examples, the text annotation may be the result of determining whether the inspection image has a lesion, such as "lesion present" or "no lesion". In some examples, the text annotation may also be the type of lesion. For example, when the inspection image is a fundus image, the text annotation may be "macular lesion", "hypertensive retinopathy" or "diabetic retinopathy".

[0168] In some examples, the training data set may be stored in storage module 431. In some examples, storage module 431 may be configured to store the training data set.

[0169] In some examples, the training data set may include 30%-60% of the inspection images with no lesion result annotation results. In some examples, the training data set may include 10%, 20%, 30%, 40%, 50% or 60% of the inspection images with no lesion result annotation results.

[0170] In some examples, the training data set can be stored using storage module 431. In some examples, storage module 431 can include memory 20.

[0171] In some examples, the storage module 431 may be configured to store the inspection image and the annotation image associated with the inspection image.

[0172] In some examples, the artificial neural network module 420 may receive a training data set stored by the storage module 431 .

[0173] In some examples, the training dataset can be preprocessed.

[0174] In step S200, the training data set may be input into the artificial neural network module 420, and a first recognition result, a second recognition result, and a third recognition result matching each inspection image may be obtained. In some examples, the training data set may be input into the artificial neural network module 420 to obtain a feature map, an attention heat map, and a complementary attention heat map. In some examples, feature extraction may be performed on the inspection image to obtain a feature map. In some examples, the feature map may be processed based on an attention mechanism to obtain an attention heat map. In some examples, the attention heat map may be processed based on a complementary attention mechanism to obtain a complementary attention heat map.

[0175] In some examples, step S200 may be implemented using processing module 432. In some examples, processing module 432 may include at least one processor 10.

[0176] In some examples, as described above, the artificial neural network module 420 may include a first artificial neural network 421 , a second artificial neural network 422 , and a third artificial neural network 423 .

[0177] In some examples, the processing module 432 can be configured to extract features from the inspection image using the first artificial neural network 421 to obtain a feature map. In some examples, the processing module 432 can be configured to obtain an attention heat map indicating a lesion area and a complementary attention heat map indicating a non-lesion area using the second artificial neural network 422.

[0178] In some examples, the processing module 432 can be configured to obtain a recognition result including tissue lesion recognition using the third artificial neural network 423. As described above, the third artificial neural network 423 can include an output layer. In some examples, the output layer can be configured to output a recognition result reflecting the inspection image. In this case, the third artificial neural network 423 can output a recognition result reflecting the inspection image.

[0179] In some examples, the processing module 432 may utilize the third artificial neural network 423 to identify the inspection image based on the feature map to obtain a first identification result.

[0180] In some examples, the processing module 432 may utilize the third artificial neural network 423 to identify the inspection image based on the feature map and the attention heat map to obtain a second identification result.

[0181] In some examples, the processing module 432 may utilize the third artificial neural network 423 to identify the inspection image based on the feature map and the complementary attention heat map to obtain a third identification result.

[0182] In some examples, the tissue lesion may be a fundus lesion. In this case, the artificial neural network module 420 can be used for fundus lesion recognition of fundus images.

[0183] In step S300 , a total loss function may be calculated based on the first recognition result, the second recognition result, and the third recognition result.

[0184] In some examples, step S300 may be implemented using the optimization module 433 .

[0185] In some examples, the optimization module 433 can obtain the total loss function of the artificial neural network module 420 based on the first loss function, the second loss function, and the third loss function. In this case, the artificial neural network module 420 can be optimized using the total loss function.

[0186] In some examples, the optimization module 433 can combine the first recognition result with the annotated image to obtain a first loss function when the attention mechanism is not used. In some examples, the first loss function can be used to evaluate the degree of inconsistency between the recognition result and the annotated result of the inspection image when the attention mechanism is not used. In this case, the accuracy of tissue lesion recognition by the artificial neural network module 420 when the attention mechanism is not used can be improved.

[0187] In some examples, the optimization module 433 can combine the second recognition result with the annotated image to obtain a second loss function when using the attention mechanism. In some examples, the second loss function can be used to evaluate the degree of inconsistency between the recognition result and the annotated result of the inspection image when using the attention mechanism. In this case, the accuracy of tissue lesion recognition by the artificial neural network module 420 when using the attention mechanism can be improved.

[0188] In some examples, the optimization module 433 can combine the third recognition result with the annotated image with the annotated result without lesions to obtain a third loss function when using the complementary attention mechanism. In some examples, the third loss function can be used to evaluate the degree of inconsistency between the recognition result of the inspection image when using the complementary attention mechanism and the recognition without lesions. In this case, the accuracy of tissue lesion recognition by the artificial neural network module 420 when using the complementary attention mechanism can be improved.

[0189] In some examples, the first loss function, the second loss function, and the third loss function can be obtained by an error loss function. In some examples, the error loss function can be a correlation function, an L1 loss function, an L2 loss function, or a Huber loss function, etc., which is used to evaluate the correlation between the true value (i.e., the labeling result) and the predicted value (i.e., the recognition result).

[0190] In some examples, the total loss function may include a first loss term, a second loss term, and a third loss term.

[0191] In some examples, the first loss term may be positively correlated with the first loss function. In this case, the first loss term can be used to evaluate the degree of inconsistency between the recognition result and the annotation result of the inspection image when the attention mechanism is not used, thereby improving the accuracy of tissue lesion recognition.

[0192] In some examples, the second loss term may be positively correlated with the difference between the second loss function and the first loss function. In some examples, when the second loss function is less than the first loss function, the second loss term may be a constant value. In this case, the second loss term can be used to evaluate the degree of inconsistency between the recognition result of the inspection image when the attention mechanism is used and the recognition result when the attention mechanism is not used.

[0193] In some examples, the second loss term may be positively correlated with the difference between the second loss function and the first loss function. Specifically, when the difference between the second loss function and the first loss function is greater than zero, the difference between the second loss function and the first loss function may be used as the second loss term, and when the difference between the second loss function and the first loss function is less than zero, the second loss term may be set to zero. In this case, the second loss term can be used to evaluate the degree of inconsistency between the first recognition result and the second recognition result, so that the second recognition result can be closer to the annotation result than the first recognition result.

[0194] In some examples, the third loss term may be positively correlated with the third loss function. In this case, the third loss term can be used to evaluate the degree of inconsistency between the third recognition result and the lesion-free annotation result of the inspection image when the complementary attention mechanism is used, thereby reducing the occurrence of misjudgment or missed judgment.

[0195] In some examples, the total loss function may further include a fourth loss term. In some examples, the fourth loss term may be a regularization term. In some examples, the fourth loss term may be a regularization term for the attention heat map. In some examples, the regularization term may be obtained based on total variation. In this case, overfitting of the artificial neural network module 420 may be suppressed.

[0196] In some examples, the total loss function may include loss item weight coefficients that match the respective loss items. In some examples, the total loss function may also include a first loss item weight coefficient that matches the first loss item, a second loss item weight coefficient that matches the second loss item, a third loss item weight coefficient that matches the third loss item, and a fourth loss item weight coefficient that matches the fourth loss item, etc.

[0197] In some examples, the first loss term may be multiplied by the first loss term weight coefficient, the second loss term may be multiplied by the second loss term weight coefficient, the third loss term may be multiplied by the third loss term weight coefficient, the fourth loss term may be multiplied by the fourth loss term weight coefficient, and the fifth loss term may be multiplied by the fifth loss term weight coefficient. Thus, the influence of each loss term on the total loss function can be adjusted by the loss term weight coefficient.

[0198] In some examples, the loss term weight coefficient can be set to 0. In some examples, the loss term weight coefficient can be set to a positive number. In this case, since each loss term is a non-negative number, the value of the total loss function can be not less than zero.

[0199] In some examples, the total loss function can be expressed as:

[0200]

[0201] Among them, L is the total loss function, λ 1 is the weight coefficient of the first loss term, λ 2 is the weight coefficient of the second loss term, λ 3 is the weight coefficient of the third loss term, λ 4 is the weight coefficient of the fourth loss term, f is the error loss function, X is the inspection image, F(X) is the feature map generated by the inspection image X after passing through the first artificial neural network 421, l(X) is the annotation result of the inspection image X, max is the maximum value function, C is a classifier function that outputs the recognition result based on the input feature map or feature combination set, margin is a preset parameter, l 0 is the annotation result without lesions, M(X) is the attention heat map matching the inspection image X, is a complementary attention heat map matched with the inspection image X, “·” in the functional formula of the total loss function is a dot multiplication operation of the matrix, and Regularize(M) is a regularization term for the attention heat map M. In some examples, the classifier function can be implemented by the third artificial neural network 423.

[0202] In some examples, the optimization module 433 can use the first loss function, the second loss function, and the third loss function to obtain a total loss function including a first loss term based on the first loss function, a second loss term based on the difference between the second loss function and the first loss function, and a third loss term based on the third loss function, and use the total loss function to optimize the artificial neural network module 420.

[0203] In some examples, the total loss function may also include a fifth loss term. In some examples, the fifth loss term may be the total area term of the attention heat map. Specifically, the total area term of the attention heat map may be the area judged as the lesion area within the attention heat map. In some examples, the total area term of the attention heat map M(X) may be represented by the formula SUM(M(X)). In some examples, the fourth loss term may be used to train the artificial neural network module 420 to reduce the lesion area within the attention heat map. In this case, the fifth loss term may be used to evaluate the area of ​​the lesion area within the attention heat map and to control the number of pixels in the attention heat map that have a greater impact on the recognition results, thereby limiting the network's attention to pixels that have a greater impact on the recognition results. Thus, the accuracy of lesion area recognition can be increased.

[0204] In some examples, the total loss function may further include a sixth loss term. In some examples, the sixth loss term may be used to evaluate the degree of inconsistency between the boxed region of the lesion region in the recognition result and the manually annotated box of the lesion region in the annotated image.

[0205] In step S400 , the artificial neural network module 420 may be optimized using a total loss function.

[0206] In some examples, step S400 may be implemented using optimization module 433 .

[0207] In some examples, the optimization module 433 can optimize the artificial neural network module 420 using the total loss function to minimize the total loss function. In this case, the total loss function can be minimized to improve the accuracy of tissue lesion recognition of the artificial neural network module 420.

[0208] In some examples, the optimization module 433 can obtain a total loss function based on the first loss term, the second loss term, the third loss term, and the total area term of the attention heat map, and optimize the artificial neural network module 420 using the total loss function to obtain an artificial neural network module 420 that can be used for tissue lesion recognition. Thus, the accuracy of tissue lesion recognition of the artificial neural network module 420 can be further improved.

[0209] In some examples, the optimization module 433 may adjust the total loss function by changing the weights of the first loss term, the second loss term, the third loss term, and the fourth loss term.

[0210] In some examples, the optimization module 433 can optimize the artificial neural network module 420 based on the first loss term and the sixth loss term as the total loss function (i.e., the loss term weight coefficients of other loss terms are set to zero). In this way, the accuracy of the attention heat map and the complementary attention heat map generated by the second artificial neural network 422 can be improved.

[0211] In some examples, during the optimization process, the loss term weight coefficient in the total loss function can be modified.

[0212] In some examples, the optimization module 433 can use an optimization algorithm to iterate the parameters in the total loss function multiple times to reduce the value of the total loss function. For example, in this embodiment, a mini-batch stochastic gradient descent algorithm can be used to randomly select a set of input function parameters and then iterate the parameters multiple times to reduce the value of the loss function.

[0213] In some examples, training is paused when the total loss function is less than a second preset value or the number of iterations exceeds three preset values.

[0214] In some examples, the optimization module 433 may pre-train the artificial neural network module 420 without using the attention mechanism, and then train the artificial neural network module 420 using the attention mechanism. In this case, the training speed can be accelerated.

[0215] In some examples, the optimization module 433 can simultaneously train the first artificial neural network 421, the second artificial neural network 422, and the third artificial neural network 423. In this case, the training speed can be accelerated.

[0216] In some examples, after training is completed, the optimization module 433 may use, for example, 0-20,000 tissue images (eg, fundus images) as test tissue images to form a test set.

[0217] In some examples, the test tissue image may be used for post-training testing of the artificial neural network module 420 .

[0218] Figure 9(a) is a schematic diagram showing an example of a lesion area of ​​a fundus image obtained without using the attention mechanism training involved in the example of the present disclosure. Figure 9(b) is a schematic diagram showing an example of a lesion area of ​​a fundus image obtained by using the complementary attention mechanism training involved in the example of the present disclosure.

[0219] In some examples, the accuracy of tissue lesion recognition of fundus images trained using complementary attention mechanisms is higher. As an example of not using the attention mechanism, Figure 9(a) shows the lesion area A of the fundus image trained without the attention mechanism. As an example of the complementary attention mechanism, Figure 9(b) shows the lesion area B of the fundus image trained using the complementary attention mechanism.

[0220] Although the present disclosure is specifically described above in conjunction with the accompanying drawings and embodiments, it is to be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the protection scope of the present disclosure.

Claims

1. A method for optimizing an artificial neural network module, wherein the artificial neural network module is used for identifying fundus lesions, characterized in that: The optimization method comprises: Prepare a training data set, wherein the training data set includes a plurality of inspection images and annotated images associated with the inspection images, wherein the annotated images include annotated results with lesions or annotated results without lesions; Inputting the training data set into an artificial neural network module, obtaining a first recognition result based on the inspection image, obtaining a second recognition result based on the inspection image and an attention heat map, and obtaining a third recognition result based on the inspection image and a complementary attention heat map, wherein the sum of two pixel values ​​corresponding to pixels at the same position in the attention heat map and the complementary attention heat map is a constant value; Obtaining a total loss function based on the first recognition result, the second recognition result, the third recognition result and the annotated image, the total loss function comprising a first loss term, a second loss term and a third loss term, the first loss term being used to evaluate the degree of inconsistency between the first recognition result and the annotated result, the second loss term being used to evaluate the degree of inconsistency between the second recognition result and the first recognition result, and the third loss term being used to evaluate the degree of inconsistency between the third recognition result and the annotated result without lesions; The artificial neural network module is trained using the total loss function.

2. The optimization method according to claim 1, characterized in that: The total loss function also includes a total area term of the attention heat map, and the total area term is used to evaluate the area of ​​the lesion area.

3. The optimization method according to claim 1, characterized in that: The total loss function also includes a regularization term for the attention heat map.

4. The optimization method according to claim 1, characterized in that: After obtaining the attention heat map and the complementary attention heat map, the attention heat map and / or the complementary attention heat map are normalized.

5. The optimization method according to claim 1, characterized in that: The total loss function includes loss item weight coefficients matched to the respective loss items.

6. The optimization method according to claim 1, characterized in that: The training mode of the artificial neural network module is weak supervision.

7. The optimization method according to claim 1, characterized in that: The total loss function satisfies the formula: Wherein, L represents the total loss function, λ1 is the weight coefficient of the first loss term, λ2 is the weight coefficient of the second loss term, λ3 is the weight coefficient of the third loss term, λ4 is the weight coefficient of the fourth loss term, f is the error loss function, X represents the inspection image, F(X) is the feature map generated by the inspection image X after passing through the first artificial neural network, l(X) is the annotation result of the inspection image X, max is the maximum value function, C is the classifier function that outputs the recognition result based on the input feature map or feature combination set, margin is a preset parameter, l0 is the annotation result without lesions, M(X) is the attention heat map matching the inspection image X, is the complementary attention heat map that matches the inspection image X, “·” is the dot multiplication operation of the matrix, and Regularize(M) is the regularization term for the attention heat map M.

8. The optimization method according to claim 1, characterized in that: The total loss function includes a sixth loss term, and the sixth loss term is used to evaluate the degree of inconsistency between the box-selected area of ​​the lesion area in the recognition result and the labeled box of the lesion area manually labeled in the labeled image.

9. The optimization method according to claim 1, characterized in that: The parameters of the artificial neural network module are iterated multiple times by means of a mini-batch stochastic gradient descent algorithm.

10. The optimization method according to claim 6 or 9, characterized in that: When the total loss function is less than a second preset value or the number of iterations exceeds three preset values, training is suspended.

Images