An image detection and recognition method, device and computer readable storage medium

Abstract

The present application belongs to the technical field of image detection and recognition, and relates to an image detection and recognition method, device and computer readable storage medium. The method comprises the following steps: taking an eye fundus image sample pair in which the labels of the left eye fundus image and the right eye fundus image are the same as a group of eye fundus image sample pairs, giving a new label to each group of eye fundus image sample pairs as the true label thereof; inputting the left and right eye fundus images of the eye fundus image sample pair into two feature extraction branches to output a first feature map and a second feature map; inputting the first feature map and the second feature map into two attention branches to output a first deep feature map and a second deep feature map; splicing the first deep feature map and the second deep feature map to input a classification module to output the predicted label of the eye fundus image sample pair, calculate the value of a detection and recognition loss function, and train the double-branch feature extraction module, the double-branch attention module and the classification module to obtain a trained eye fundus image detection and recognition model.

Description

Technical Field

[0001] The present invention relates to the field of image detection and recognition technology, and in particular to an image detection and recognition method, device and computer-readable storage medium. Background Art

[0002] The eye is the primary organ for acquiring external information. Abnormalities can lead to visual impairment and impact quality of life. Traditional fundus abnormality testing relies on manual analysis of fundus images to determine the presence and nature of abnormalities. With the continuous advancement of deep learning, an increasing number of neural networks are being used for fundus image analysis and recognition, gradually improving the efficiency and accuracy of fundus abnormality detection and recognition.

[0003] In the prior art, when performing fundus abnormality detection and identification, fundus images are subjected to feature extraction and classification by constructing convolutional neural networks with various simple or complex structures. However, due to the large number of types of fundus abnormalities, fundus images with different abnormalities exhibit different characteristics. At the same time, there may be multiple abnormality categories in one fundus image, which makes the fundus abnormality detection and identification problem more complicated. When faced with fundus images with multiple abnormal features, convolutional neural networks find it difficult to accurately identify various abnormality categories and make correct prediction results. In addition, since fundus abnormalities usually have bilateral correlation characteristics, the prior art only extracts features from unilateral fundus images, ignoring the effective feature information that may exist in the other side of the fundus image, which also leads to low accuracy of fundus abnormality detection and identification results.

[0004] In summary, existing fundus abnormality detection and recognition methods have the problem of low accuracy of detection and recognition results. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low accuracy of detection and identification results in the prior art fundus abnormality detection and identification methods.

[0006] To solve the above technical problems, the present invention provides an image detection and recognition method, comprising:

[0007] Obtain fundus image sample pairs containing left and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; treat fundus image sample pairs with the same labels for the left and right fundus images as a group of fundus image sample pairs, and assign a new label to each group of fundus image sample pairs as its true label;

[0008] Inputting the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the dual-branch feature extraction module respectively, and outputting a first feature map of the left fundus image and a second feature map of the right fundus image;

[0009] The first feature map and the second feature map are respectively input into two attention branches in the double-branch attention module, and a first depth feature map of a left fundus image and a second depth feature map of a right fundus image are output;

[0010] The first depth feature map and the second depth feature map are spliced and input into the classification module, and a predicted label of the fundus image sample pair is output, and a value of a detection and recognition loss function is calculated based on the predicted label and a real label thereof;

[0011] The double-branch feature extraction module, the double-branch attention module and the classification module are iteratively trained until the value of the detection and recognition loss function is minimum, and a trained fundus image detection and recognition model is obtained.

[0012] Preferably, the first depth feature map and the second depth feature map are spliced and input into the classification module, and a predicted label of the fundus image sample pair is output, comprising:

[0013] The first depth feature map and the second depth feature map are spliced to obtain a spliced feature map;

[0014] Global spatial information compression is performed on the first depth feature map, the second depth feature map and the spliced feature map by using a global average pooling module, and a first feature vector, a second feature vector and a global feature vector are obtained;

[0015] The first feature vector is input into a first fully connected layer, and a predicted label of the left fundus image is output; the second feature vector is input into a second fully connected layer, and a predicted label of the right fundus image is output; and the global feature vector is input into a third fully connected layer, and a predicted label of the fundus image sample pair is output.

[0016] Preferably, the value of the detection and recognition loss function is calculated based on the predicted label and the real label thereof, comprising:

[0017] A value of a global loss function is calculated based on a logarithmic loss between the predicted label of the fundus image sample pair and the real label thereof;

[0018] A value of a multi-label loss function is obtained based on a sum of a binary cross-entropy loss of the predicted label of the left fundus image and a label thereof and a binary cross-entropy loss of the predicted label of the right fundus image and a label thereof;

[0019] A first difference value between the predicted label of the left fundus image and the predicted label of the right fundus image is calculated, a second difference value between the label of the left fundus image and the label of the right fundus image is calculated, and a value of a difference consistency loss function is calculated based on a binary cross-entropy loss of the first difference value and the second difference value;

[0020] The value of the global loss function, the value of the multi-label loss function and the value of the difference consistency loss function are weighted and summed to obtain the value of the detection and recognition loss function.

[0021] Preferably, the calculation formula of the global loss function is:

[0022] ,

[0023] in, represents the global loss function; ∈ ; K represents the total number of true labels after label conversion, Predict the probability that the fundus image sample pair belongs to the kth category; is the true probability that the fundus image sample pair belongs to the kth category;

[0024] The calculation formula of the multi-label loss function is:

[0025] ,

[0026] in, represents the multi-label loss function; ∈ , N represents the total number of labels before label conversion, ∈ ; represents the predicted probability that the left fundus image belongs to the i-th category; represents the true probability that the left fundus image belongs to the i-th category; represents the predicted probability that the right fundus image belongs to the i-th category; represents the true probability that the right fundus image belongs to the i-th category;

[0027] The calculation formula of the difference consistency loss function is:

[0028] ,

[0029] in, represents the difference consistency loss function; , represents the first difference in the predicted probability that the left and right fundus images belong to the i-th class, The second difference value representing the true probability of the left and right fundus images belonging to the i-th class;

[0030] The calculation formula of the detection and recognition loss function is:

[0031] ,

[0032] in, Represents the detection and recognition loss function; express The weight parameter of express The weight parameter of a weight parameter of the attention module.

[0033] Preferably, the two attention branches in the double-branch attention module are different, and the first feature map and the second feature map are respectively input into the two attention branches in the double-branch attention module, comprising:

[0034] the first feature map is input into the first attention branch structure, and the second feature map is input into the second attention branch structure; or,

[0035] the first feature map is input into the second attention branch structure, and the second feature map is input into the first attention branch structure.

[0036] Preferably, the step of the first attention branch in the double-branch attention module performing deep feature extraction on the input feature map comprises:

[0037] a plurality of first convolution kernels with the same number of channels as the input feature map are used to respectively convolve each channel of the input feature map, and a plurality of feature maps with the same number of channels as the input feature map are output; a second convolution kernel with the same size as the channel number of the input feature map is used to weight and fuse the plurality of feature maps to obtain a first fused feature map;

[0038] a plurality of first pointwise convolution kernels are used to perform a plurality of pointwise convolution operations on the input feature map, and a second fused feature map is obtained based on the output of the last first pointwise convolution kernel;

[0039] the first fused feature map and the second fused feature map are element-wise pointwise added and fused, and a deep feature map is output.

[0040] Preferably, the step of the second attention branch in the double-branch attention module performing deep feature extraction on the input feature map comprises:

[0041] a plurality of second pointwise convolution kernels are used to perform a plurality of pointwise convolution operations on the input feature map, and a third fused feature map is obtained based on the output of the last second pointwise convolution kernel;

[0042] the cross-space multi-scale attention module is used to adaptively recalibrate the weight of each channel of the input feature map, and a global feature map containing the weight information of each channel is output;

[0043] the third fused feature map and the global feature map are element-wise pointwise added and fused, and a deep feature map is output.

[0044] Preferably, before the left fundus image and the right fundus image in the fundus image sample pair are input into the double-branch feature extraction module, the method further comprises:

[0045] ​The gray scale histogram of the left fundus image and the right fundus image is obtained, if the pixel distribution interval width in the gray scale histogram is less than a preset width, and / or the number of pixels in the high gray scale area or the low gray scale area in the gray scale histogram is greater than or equal to a preset number, the gray scale histogram is taken as a gray scale histogram to be repaired;

[0046] The histogram equalization method is used to optimize the gray scale histogram to be repaired, and the optimized fundus image is obtained based on the optimized gray scale histogram to be repaired.

[0047] The application further provides an image detection and recognition device, comprising:

[0048] The data acquisition and label conversion module is used for acquiring the fundus image sample pairs in the training set and the labels thereof, wherein the fundus image sample pairs comprise left fundus images and right fundus images, and the labels are normal or abnormal categories of the fundus images; the fundus image sample pairs with the same labels of the left fundus images and the right fundus images are taken as a group of fundus image sample pairs, and a new label is given to each group of fundus image sample pairs as a real label thereof;

[0049] The global feature extraction module is used for inputting the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the double-branch feature extraction module respectively, and outputting a first feature map of the left fundus image and a second feature map of the right fundus image;

[0050] The deep feature extraction module is used for inputting the first feature map and the second feature map into two attention branches of the double-branch attention module respectively, and outputting a first deep feature map of the left fundus image and a second deep feature map of the right fundus image;

[0051] The category prediction and loss function calculation module is used for splicing the first deep feature map and the second deep feature map and inputting the first deep feature map and the second deep feature map into the classification module, outputting a predicted label of the fundus image sample pair, and calculating a value of a detection and recognition loss function based on the predicted label and the real label;

[0052] The model training and acquisition module is used for iteratively training the double-branch feature extraction module, the double-branch attention module and the classification module until the value of the detection and recognition loss function is minimum, and obtaining a trained fundus image detection and recognition model.

[0053] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the image detection and recognition method.

[0054] The image detection and recognition method provided by the application has the following beneficial effects:

[0055] 1、Firstly, the multi-classification problem is converted into a single classification problem, that is, a new label is created for each group of fundus image sample pairs by identifying the normal or abnormal class combination contained in the labels of all fundus image sample pairs in the training set, so that each fundus image sample pair in the training set corresponds to a label, rather than the label of the left fundus image and the label of the right fundus image, which enables the model to learn the feature information of a single class and improves the accuracy of fundus image detection and recognition. Further, the application considers that the feature information of the left and right fundus images has relevance, but if the same feature extraction network is used to extract the features of the two images, the bilateral features extracted may interfere with each other. Therefore, the application designs a double-branch feature extraction module and a double-branch attention module based on a double-branch architecture to fully extract the feature information of the left and right fundus images, and fuse the feature information of the left and right fundus images before classification. Finally, the prediction result of the abnormal detection and recognition of the fundus image sample pair is directly output, the symmetry and relevance between the bilateral structures of the fundus are considered, and the detection and recognition loss is calculated based on the prediction result and the converted label, so as to train the model. Therefore, the scheme provided by the application not only solves the problem that the multi-class label confusion of the fundus abnormal detection task causes the model to be unable to effectively learn the single-class feature, but also avoids the information loss problem of single-side fundus image detection and recognition, thereby improving the precision of fundus abnormal detection and recognition.

[0056] 2、In constructing the detection and recognition loss function, in addition to considering the prediction loss between the prediction label based on the bilateral fusion feature and the true label thereof, the left and right fundus images are also classified and predicted respectively, and a multi-label independent prediction loss is constructed based on the predicted class of the left fundus image and its normal or abnormal class, and the predicted class of the right fundus image and its normal or abnormal class. And considering the consistency and difference features between the left and right eyes, a difference consistency loss is constructed based on the difference between the prediction labels of the left and right fundus images and the difference between the normal or abnormal classes of the left and right fundus images. Both the bilateral fusion feature and the independent feature of the left and right fundus images are considered, and the alignment of the predicted difference and the true difference of the left and right eyes is also considered, so as to match the single classification task and the bilateral difference recognition task in fundus abnormal recognition, and further improve the accuracy of the fundus image detection and recognition result.

[0057] 3、The two attention branches in the double-branch attention module in the application adopt different structures. The first attention branch structure is used for local feature extraction, which reduces the amount of calculation while strengthening the sensitivity to local key structures (such as blood vessels, texture details, etc.); the second attention branch structure is used for global context feature extraction, which captures global context dependence through cross-dimension interaction, adaptively highlights the channels related to abnormal features (such as deep blood vessel features related to glaucoma), and suppresses irrelevant channels, establishing semantic consistency association between the left and right eye features; through two attention branches with different structures, local-global complementarity can be achieved. Branch one captures monocular local lesion details, and branch two establishes global consistency association between the two eyes. The two are spliced to form a fusion feature with local detail enhancement and global semantic alignment. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to make the content of the application more easily understood, the application will be further described in detail below according to specific embodiments of the application and in conjunction with the drawings, in which:

[0059] Figure 1 The image detection and recognition method flowchart provided for the application;

[0060] Figure 2 The image preprocessing flowchart provided for the application;

[0061] Figure 3 The fundus image schematic diagram before and after background cropping provided for the application; wherein, Figure 3 (a) in the above (a) is the fundus image before background cropping, Figure 3 (b) in the above (b) is the fundus image after background cropping;

[0062] Figure 4 The fundus image schematic diagram before and after histogram equalization provided for the application; wherein, Figure 4 (a) in the above (a) is the fundus image before histogram equalization, Figure 4 (b) in the above (b) is the fundus image after histogram equalization;

[0063] Figure 5 The label conversion principle schematic diagram provided for the application;

[0064] Figure 6 The image detection and recognition model architecture schematic diagram provided for the application;

[0065] Figure 7 The cross-space multi-scale attention module structure schematic diagram provided for the application;

[0066] Figure 8 The image detection and recognition device structure schematic diagram provided for the application. DETAILED DESCRIPTION

[0067] The application will be further described below in conjunction with the drawings and specific embodiments so that those skilled in the art can better understand and implement the application, but the embodiments are not intended to limit the application.

[0068] Please refer to Figure 1 , Figure 1 The image detection and recognition method provided by the application is shown in the flowchart, and the method specifically includes:

[0069] S10: Obtain an eye fundus image sample pair in the training set and its label, and the label is the normal or abnormal category of the eye fundus image; the eye fundus image sample pair with the same label of the left eye fundus image and the right eye fundus image is taken as a group of eye fundus image sample pairs, and a new label is given to each group of eye fundus image sample pairs as its real label.

[0070] Specifically, each eye fundus image sample pair includes a left eye fundus image and a right eye fundus image belonging to the same object, and the label of each eye fundus image sample pair includes the normal or abnormal category of the left eye fundus image and the normal or abnormal category of the right eye fundus image; for example, if the left eye fundus image in an eye fundus image sample pair is normal, and the right eye fundus image is determined to be A type abnormality and B type abnormality at the same time, then the label of the eye fundus image sample pair is left eye fundus image normal, right eye fundus image A type abnormality + B type abnormality.

[0071] S20: The left eye fundus image and the right eye fundus image in the eye fundus image sample pair are respectively input into two feature extraction branches of the double-branch feature extraction module, and the first feature map of the left eye fundus image and the second feature map of the right eye fundus image are output.

[0072] S30: The first feature map and the second feature map are respectively input into two attention branches in the double-branch attention module, and the first deep feature map of the left eye fundus image and the second deep feature map of the right eye fundus image are output.

[0073] S40: The first deep feature map and the second deep feature map are spliced and input into the classification module, the predicted label of the eye fundus image sample pair is output, and the value of the detection and recognition loss function is calculated based on the predicted label and the real label.

[0074] S50: The double-branch feature extraction module, the double-branch attention module and the classification module are iteratively trained until the value of the detection and recognition loss function is minimum, and a trained eye fundus image detection and recognition model is obtained.

[0075] Due to the great difference in illumination conditions, image resolution, contrast, color saturation and the like between different fundus images, if there is a difference between the fundus image samples in the training set or between the left and right fundus images in the fundus image sample pair, the recognition and feature extraction accuracy of the detection recognition model for the images will be inconsistent, which will affect the accuracy of the fundus image detection recognition result. Therefore, some embodiments of the present application eliminate these differences by preprocessing the fundus image sample pair.

[0076] As shown in Figure 2 is an image preprocessing flowchart provided by the present application, which specifically includes background cropping, histogram equalization, filtering denoising and data enhancement.

[0077] Specifically, the fundus image will contain a large area of black background, which is not substantially helpful for fundus anomaly detection and recognition, and is equivalent to noise in the fundus image, which will reduce the signal-to-noise ratio of the fundus image and increase the difficulty of the model to extract effective information in the fundus image. Therefore, the black background area of the fundus image can be cropped.

[0078] For example, the pixels of the fundus image can be adjusted to 224*224, as shown in Figure 3 is a schematic diagram of the fundus image before and after background cropping; wherein, Figure 3 (a) in the above figure is the fundus image before background cropping, Figure 3 (b) in the above figure is the fundus image after background cropping. By comparing Figure 3 (a) and (b) in the above figure, it can be seen that the cropped fundus image focuses more on the effective information area.

[0079] Further, due to the difference in imaging conditions, some fundus images may present overall brightness abnormalities (overexposure or underexposure), and the color of the blood vessel structure and the optic disc area is similar, which will also affect the accuracy of subsequent feature extraction and recognition. Therefore, in order to optimize the quality of the fundus image and improve the recognizability of the fundus anomaly feature, the fundus image can be subjected to contrast enhancement by histogram equalization.

[0080] Specifically, the present application further includes the following steps before step S20:

[0081] Obtaining the gray histogram of the left fundus image and the right fundus image, if the pixel distribution interval width in the gray histogram is less than a preset width, and / or the number of pixels in the high gray area or low gray area in the gray histogram is greater than or equal to a preset number, the gray histogram is taken as a gray histogram to be repaired.

[0082] Optimizing the gray histogram to be repaired by using the histogram equalization method, and mapping the optimized fundus image based on the optimized gray histogram to be repaired.

[0083] The overexposure of the fundus image is that a large number of pixel points in the gray histogram are concentrated in the high gray area (pixel value 180~255 area), and the underexposure of the fundus image is that a large number of pixel points in the gray histogram are concentrated in the low gray area (pixel value 0~50 area), and the insufficient contrast of the fundus image is that the pixel points in the gray histogram are distributed in a narrow interval and concentrated in a narrow interval, which will lead to the weakening of the gray difference between the fundus tissue structures, forming a low-contrast fundus image. Therefore, the quality of the fundus image is reflected by analyzing the gray histogram, and the gray histogram of the fundus image with low quality is improved by histogram equalization, so that the fundus image with high quality is obtained by mapping the optimized gray histogram.

[0084] As a classic nonlinear contrast enhancement algorithm, histogram equalization (HE) can effectively improve the low-quality fundus image by optimizing the histogram distribution through dynamic remapping of gray levels. The originally concentrated distributed gray values are remapped to a wider dynamic range, thereby obtaining an approximately uniform histogram distribution. Not only can it correct the gray shift caused by abnormal illumination, but also can enhance the gray difference between the blood vessel network and the optic disc area, thereby establishing a good visual basis for subsequent feature extraction and recognition. Figure 4 Fig. 1 shows the fundus images before and after histogram equalization provided by the present application; wherein, Figure 4 (a) in the fundus image before histogram equalization, Figure 4 (b) in the fundus image after histogram equalization, from Figure 4 It can be seen from the fundus image after histogram equalization that the blood vessel pattern is clearer, and the contrast is also significantly enhanced.

[0085] Further, since there are many types of fundus abnormalities, i.e., the fundus abnormality detection is a multi-classification problem, the complexity is high, so that the model needs to learn the association between multiple abnormal categories. When there are multiple categories of abnormal problems in the fundus, the model needs to accurately predict all the abnormal categories, which to some extent increases the complexity of fundus abnormality recognition, and further affects the accuracy of model detection and recognition.

[0086] Based on the above reasons, step S10 of the present application first converts the multi-classification problem into a single-classification problem, that is, a new label is created for each group of fundus image sample pairs by identifying the abnormal class combinations contained in the labels of all fundus image sample pairs in the training set. For example, if the fundus images in the training set include normal and 7 abnormal classes, there are 255 class combinations considering only the combinations between classes, but in actual application, some classes have mutual exclusivity, for example, fundus images cannot be both normal and abnormal, so the real class combinations are much less than 255, it is worth noting that the present application finally obtains 37 different class combinations by traversing the disclosed fundus image sample data set.

[0087] Specifically, based on the above principle, the present application utilizes the mutual exclusivity of fundus image labels to classify all fundus image sample pairs and assign the same new label to each fundus image sample pair in each group of fundus image sample pairs. For example, as shown in Figure 5 The label conversion principle provided by the present application is shown in the schematic diagram, and the class combination and the assignment of the new label means that the real label of each fundus image sample pair is determined according to the label of each fundus image sample pair, if a fundus image sample pair simultaneously exists C class abnormality and O class abnormality, then its new label is the label representing the combination of C and O.

[0088] Specifically, the present application converts the labels so that each fundus image sample pair in the training set corresponds to one class, so that the model can learn the feature information of a single class, thereby improving the accuracy of fundus image detection and recognition.

[0089] Further, since the feature information of left and right fundus images has mutual relevance when detecting and recognizing fundus images, the present application considers the timeliness of detection and recognition when constructing the detection and recognition model, and directly uses a double-branch feature extraction and fusion architecture, as shown in Figure 6 The detection and recognition model architecture constructed by the present application is shown in the schematic diagram, wherein the double-branch feature extraction module includes two parallel feature extraction branches, which respectively perform global feature extraction on the left fundus image and the right fundus image in a group of fundus image samples, then two attention branches in the double-branch attention module are used to extract deep features from the left fundus image and the right fundus image, fully mining the potential complementary information, improving the perception ability of the model for abnormal regions, and finally the features of the left and right fundus images are fused, and a classification model is used to complete the classification task.

[0090] Specifically, the two feature extraction branches in the dual-branch feature extraction module are independent and identical in structure. The feature extraction network can be selected from mainstream convolutional neural network models pre-trained on the ImageNet dataset, such as ResNet, DenseNet, and MobileNet, etc. These models can effectively learn discriminative feature representations from large-scale fundus images by stacking convolutional layers, pooling layers, and fully connected layers. For example, ResNet alleviates the gradient vanishing problem in deep networks by introducing a residual connection mechanism, which facilitates the capture of high-level semantic information of images. DenseNet promotes feature transmission and reuse through inter-layer dense connections, improving the model's expression ability. MobileNet effectively balances the depth, width, and resolution of the model, and uses depthwise separable convolution to significantly reduce computational complexity.

[0091] During the research process, various feature extraction networks were compared and evaluated. The results showed that MobileNet_V2 achieved a better balance between accuracy and model complexity. Therefore, as a preferred embodiment, the feature extraction network is a lightweight MobileNet_V2.

[0092] Further, considering the characteristics of the dual-branch structure, we input the features of the left and right eye images into two independent attention modules, named attention1 and attention2, respectively, to enhance their response ability to key areas and suppress redundant or noisy information. Among them, one branch uses an attention mechanism centered on depthwise separable convolution (DSConv) and pointwise convolution (PConv). DSConv decomposes the standard convolution into channel-wise convolution and pointwise convolution, significantly reducing the number of parameters and computational cost while maintaining strong expression ability. The subsequent PConv outputs are processed by batch normalization (BN) to improve training stability and by ReLU activation function to enhance the non-linear modeling ability of the network. The other branch introduces an efficient multi-scale attention mechanism. This module divides the channel dimension into multiple subspaces and reconstructs it to the batch dimension, reducing the computational burden while preserving the full-channel feature information. The EMA module uses global context information to adaptively recalibrate the channel weights and enhances the modeling ability of pixel-level semantic relationships through cross-dimensional information interaction. Finally, the outputs of the two attention branches are connected in the channel dimension, realizing efficient fusion of left and right eye image information and providing more rich feature representations for the subsequent classification module.

[0093] Specifically, the two attention branch structures in the double-branch attention module are different: the first attention branch structure is used for local feature extraction, which adopts DSConv and PConv to reduce the amount of calculation while enhancing the sensitivity to local key structures (such as blood vessels, texture details, etc.); the second attention branch structure is used for global context feature extraction, which captures global context dependencies through cross-dimensional interaction, adaptively highlights channels related to abnormal features (such as deep blood vessel features related to glaucoma), and suppresses irrelevant channels, establishing semantic consistency association between left and right eye features; through two structurally different attention branches, local-global complementarity can be achieved: branch one captures monocular local lesion details, and branch two establishes bilateral global consistency association, and the two are spliced to form a fusion feature of local detail enhancement + global semantic alignment.

[0094] Further, the first feature map and the second feature map are input into the two attention branches in the double-branch attention module, including: inputting the first feature map into the first attention branch structure and inputting the second feature map into the second attention branch structure; or inputting the first feature map into the second attention branch structure and inputting the second feature map into the first attention branch structure.

[0095] The present application does not limit which attention branch is used for deep feature extraction of the first feature map and the second feature map, and in actual use, the corresponding attention branch can be selected by pre-analyzing the characteristics of the left and right fundus images.

[0096] Further, the step of the first attention branch in the double-branch attention module performing deep feature extraction on the input feature map includes:

[0097] A plurality of first convolution kernels with the same number of channels as the input feature map are used to respectively convolve each channel of the input feature map, outputting a plurality of feature maps with the same number of channels as the input feature map; a second convolution kernel with the same size as the input feature map is used to perform weighted fusion on the plurality of feature maps to obtain a first fused feature map;

[0098] A plurality of first pointwise convolution kernels are used to perform multiple pointwise convolution operations on the input feature map, and a second fused feature map is obtained based on the output of the last first pointwise convolution kernel;

[0099] The first fused feature map and the second fused feature map are element-wise pointwise added and fused, and a deep feature map is output.

[0100] The step of the second attention branch in the double-branch attention module performing deep feature extraction on the input feature map includes:

[0101] The input feature map is subjected to multiple point-by-point convolution operations by multiple second point-by-point convolution kernels in series, and a third fusion feature map is obtained based on the output of the last second point-by-point convolution kernel;

[0102] The cross-space multi-scale attention module is used to adaptively recalibrate the channel weights of the input feature map, and a global feature map containing channel weight information is output.

[0103] The third fusion feature map and the global feature map are subjected to element-level point-by-point addition fusion, and a deep feature map is output.

[0104] The cross-space multi-scale attention module is an existing module, as shown in Figure 7 The structure of the cross-space multi-scale attention module is shown in the figure.

[0105] Further, when detecting and identifying based on the deep feature map, in addition to considering the spliced deep feature map fused with left and right eye feature information, the first deep feature map corresponding to the left fundus image and the second deep feature map corresponding to the right fundus image are also detected and identified respectively, so that the symmetry and difference features between the left and right fundus images can be further mined, and the detection and identification accuracy is improved.

[0106] Specifically, the first deep feature map and the second deep feature map are spliced and input into a classification module to output a predicted label of the fundus image sample pair, including:

[0107] The first deep feature map and the second deep feature map are spliced to obtain a spliced feature map.

[0108] The global average pooling module is used to compress the global spatial information of the first deep feature map, the second deep feature map and the spliced feature map respectively to obtain a first feature vector, a second feature vector and a global feature vector.

[0109] The first feature vector is input into a first fully connected layer to output a predicted label of the left fundus image, the second feature vector is input into a second fully connected layer to output a predicted label of the right fundus image, and the global feature vector is input into a third fully connected layer to output a predicted label of the fundus image sample pair.

[0110] Specifically, the global average pooling module compresses the previously extracted feature map in the spatial dimension to generate a feature vector with global semantic information, which not only retains the key features of the fundus image, but also greatly reduces the number of parameters and avoids the risk of overfitting. Then, the feature vectors are further abstracted and converted by the fully connected layer to meet the needs of the classification task, and the predicted probabilities of the fundus image sample pair belonging to each class and the probabilities of the left and right fundus images belonging to the normal or abnormal class are output.

[0111] Further, the application carries out label conversion, and different attention branches are used to extract differential deep features of left and right fundus images, and a detection and recognition loss function is constructed based on the consistency and difference of left and right eyes to match single classification tasks and double eye differential detection and recognition tasks.

[0112] Specifically, the value of the detection and recognition loss function is calculated based on the predicted label and the real label, including:

[0113] Based on the logarithmic loss between the predicted label and the real label of the fundus image sample pair, the value of the global loss function is calculated, and the specific calculation formula is:

[0114] ,

[0115] Wherein, represents the global loss function; ∈ ; K represents the total number of real labels after label conversion, is the predicted probability that the fundus image sample pair belongs to the kth class; is the real probability that the fundus image sample pair belongs to the kth class.

[0116] Based on the sum of the binary cross-entropy loss of the predicted label of the left fundus image and its label and the binary cross-entropy loss of the predicted label of the right fundus image and its label, the value of the multi-label loss function is obtained, and the specific calculation formula is:

[0117] ,

[0118] Wherein, represents the multi-label loss function; ∈ , N represents the total number of labels before label conversion, ∈ ; represents the predicted probability that the left fundus image belongs to the ith class; represents the real probability that the left fundus image belongs to the ith class; represents the predicted probability that the right fundus image belongs to the ith class; represents the real probability that the right fundus image belongs to the ith class.

[0119] The first difference between the predicted label of the left fundus image and the predicted label of the right fundus image is calculated, the second difference between the label of the left fundus image and the label of the right fundus image is calculated, and the value of the difference consistency loss function is calculated based on the binary cross-entropy loss of the first difference and the second difference, and the specific calculation formula is:

[0120] ,

[0121] wherein, represents a difference consistency loss function; , represents a first difference value of a predicted probability that the left and right fundus images belong to the i-th class, represents a second difference value of a true probability that the left and right fundus images belong to the i-th class.

[0122] The value of the global loss function, the value of the multi-label loss function, and the value of the difference consistency loss function are weighted and summed to obtain the value of the detection and recognition loss function, and the specific calculation formula is:

[0123] ,

[0124] wherein, represents a detection and recognition loss function; represents a weight parameter of ; represents a weight parameter of ; represents a weight parameter of .

[0125] Based on the image detection and recognition method provided in the above embodiment, the embodiment of the present application further provides an image detection and recognition device, as shown in Figure 8 , the device specifically comprises:

[0126] The data acquisition and label conversion module 10 is configured to acquire a fundus image sample pair in the training set and its label, wherein the fundus image sample pair comprises a left fundus image and a right fundus image, and the label is a normal or abnormal category of the fundus image; the fundus image sample pair in which the label of the left fundus image and the label of the right fundus image are the same is taken as a group of fundus image sample pairs, and a new label is given to each group of fundus image sample pairs as its true label.

[0127] The global feature extraction module 20 is configured to input the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the double-branch feature extraction module respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image.

[0128] The deep feature extraction module 30 is configured to input the first feature map and the second feature map into two attention branches in the double-branch attention module respectively, and output a first deep feature map of the left fundus image and a second deep feature map of the right fundus image.

[0129] The category prediction and loss function calculation module 40 is configured to splice the first deep feature map and the second deep feature map and input them into the classification module, output a predicted label of the fundus image sample pair, and calculate the value of the detection and recognition loss function based on the predicted label and the true label.

[0130] The model training and obtaining module 50 is configured to iteratively train the double-branch feature extraction module, the double-branch attention module, and the classification module until the value of the detection and recognition loss function is minimized, and to obtain the trained fundus image detection and recognition model.

[0131] The embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the image detection and recognition method.

[0132] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer readable storage media containing computer usable program codes, including but not limited to disk memory, CD-ROM, optical memory, etc.

[0133] The present application is described with reference to flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0134] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0135] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable data processing device to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing device provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1steps of the functions specified in the one or more blocks.

[0136] Obviously, the above-mentioned embodiments are only examples for clearly illustrating the present application, and are not intended to limit the present application. Based on the above-mentioned embodiments, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary or possible to enumerate all the embodiments. The obvious changes or variations derived from the above-mentioned embodiments are still within the protection scope of the present application.

Claims

1. An image detection and recognition method, characterized by, The method comprises the following steps: Obtain a pair of fundus image samples and their labels in the training set, which include left and right fundus images, and the labels are normal or abnormal categories of the fundus images; Take the pair of fundus image samples with the same labels of the left and right fundus images as a group of fundus image samples, and give a new label as the true label of each group of fundus image samples; Input the left and right fundus images in the pair of fundus image samples into two feature extraction branches of a double-branch feature extraction module respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image; Input the first and second feature maps into two attention branches in a double-branch attention module respectively, and output a first deep feature map of the left fundus image and a second deep feature map of the right fundus image; Concatenate the first and second deep feature maps and input them into a classification module to output the predicted label of the pair of fundus image samples, and calculate the value of the detection and recognition loss function based on the predicted label and the true label, which specifically comprises: Calculate the value of the global loss function based on the logarithmic loss between the predicted label and the true label of the pair of fundus image samples; Calculate the value of the multi-label loss function based on the sum of the binary cross-entropy loss of the predicted label of the left fundus image and its label and the binary cross-entropy loss of the predicted label of the right fundus image and its label; Calculate the first difference value between the predicted label of the left fundus image and the predicted label of the right fundus image, calculate the second difference value between the label of the left fundus image and the label of the right fundus image, and calculate the value of the difference consistency loss function based on the binary cross-entropy loss of the first difference value and the second difference value; Weighted sum the value of the global loss function, the value of the multi-label loss function and the value of the difference consistency loss function to obtain the value of the detection and recognition loss function; Iteratively train the double-branch feature extraction module, the double-branch attention module and the classification module until the value of the detection and recognition loss function is minimized to obtain the trained fundus image detection and recognition model.

2. The image detection and recognition method of claim 1, wherein, Concatenate the first and second deep feature maps and input them into a classification module to output the predicted label of the pair of fundus image samples, which comprises: Concatenate the first and second deep feature maps to obtain a concatenated feature map; Use a global average pooling module to compress the global spatial information of the first deep feature map, the second deep feature map and the concatenated feature map to obtain a first feature vector, a second feature vector and a global feature vector; Input the first feature vector into a first fully connected layer to output the predicted label of the left fundus image, input the second feature vector into a second fully connected layer to output the predicted label of the right fundus image, and input the global feature vector into a third fully connected layer to output the predicted label of the pair of fundus image samples.

3. The image detection and recognition method of claim 1, wherein, The calculation formula of the global loss function is: ， wherein, represents a global loss function; ∈ ; K represents the total number of real labels after label conversion, is the predicted probability that the fundus image sample pair belongs to the kth category; is the real probability that the fundus image sample pair belongs to the kth category. The calculation formula of the multi-label loss function is: ， wherein, denotes a multi-label loss function; ∈ N denotes the total number of labels before label conversion, ∈ ; denotes the predicted probability that the left fundus image belongs to the i-th class; denotes the true probability that the left fundus image belongs to the i-th class; denotes the predicted probability that the right fundus image belongs to the i-th class; denotes the true probability that the right fundus image belongs to the i-th class; The calculation formula of the difference consistency loss function is: ， wherein, represents a difference consistency loss function; , represents a first difference of the predicted probability that the left and right fundus images belong to the i-th class, represents a second difference of the true probability that the left and right fundus images belong to the i-th class; The calculation formula of the detection and recognition loss function is: ， wherein, represents a detection recognition loss function; represents a weight parameter of represents a weight parameter of represents a weight parameter of 4. The image detection and recognition method of claim 1, wherein, The two attention branches in the double-branch attention module have different structures, and the first and second feature maps are input into the two attention branches in the double-branch attention module, which comprises: The first feature map is input into the first attention branch structure, and the second feature map is input into the second attention branch structure; or, The first feature map is input into the second attention branch structure, and the second feature map is input into the first attention branch structure.

5. The image detection and recognition method of claim 1, wherein, The step of performing deep feature extraction on the input feature map in the first attention branch of the double-branch attention module comprises: a plurality of first convolution kernels with the same number of channels as the input feature map are used to respectively perform convolution on each channel of the input feature map, and a plurality of feature maps with the same number of channels as the input feature map are output; a second convolution kernel with the same size as the channel number of the input feature map is used to perform weighted fusion on the plurality of feature maps, and a first fused feature map is obtained; a plurality of first point-by-point convolution kernels are used to perform a plurality of point-by-point convolution operations on the input feature map, and a second fused feature map is obtained based on the output of the last first point-by-point convolution kernel; element-level point-by-point addition fusion is performed on the first fused feature map and the second fused feature map, and a deep feature map is output.

6. The image detection and recognition method of claim 1, wherein, The step of performing deep feature extraction on the input feature map in the second attention branch of the double-branch attention module comprises: a plurality of second point-by-point convolution kernels are used to perform a plurality of point-by-point convolution operations on the input feature map, and a third fused feature map is obtained based on the output of the last second point-by-point convolution kernel; a cross-space multi-scale attention module is used to adaptively recalibrate the weight of each channel of the input feature map, and a global feature map containing the weight information of each channel is output; element-level point-by-point addition fusion is performed on the third fused feature map and the global feature map, and a deep feature map is output.

7. The image detection and recognition method of claim 1, wherein, Before the left fundus image and the right fundus image in the fundus image sample pair are input into the double-branch feature extraction module, the method further comprises: obtaining a gray histogram of the left fundus image and the right fundus image, and if the pixel distribution interval width in the gray histogram is less than a preset width, and / or the number of pixels in the high gray area or low gray area in the gray histogram is greater than or equal to a preset number, the gray histogram is taken as a repaired gray histogram; the repaired gray histogram is optimized by using a histogram equalization method, and an optimized fundus image is obtained based on the optimized repaired gray histogram.

8. An image detection and recognition apparatus, characterized by comprising: comprises: a data acquisition and label conversion module, configured to acquire fundus image sample pairs in a training set and labels of the fundus image sample pairs, wherein each fundus image sample pair comprises a left fundus image and a right fundus image, and the label is a normal or abnormal category of the fundus image; and the fundus image sample pairs in which the label of the left fundus image and the label of the right fundus image are the same are taken as a group of fundus image sample pairs, and a new label is given to each group of fundus image sample pairs as a real label of the group of fundus image sample pairs; a global feature extraction module, configured to input the left fundus image and the right fundus image in each fundus image sample pair into two feature extraction branches of a double-branch feature extraction module respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image; a deep feature extraction module, configured to input the first feature map and the second feature map into two attention branches of a double-branch attention module respectively, and output a first deep feature map of the left fundus image and a second deep feature map of the right fundus image. The category prediction and loss function calculation module is configured to splice the first deep feature map and the second deep feature map and input the spliced deep feature map into the classification module, output a predicted label of the fundus image sample pair, and calculate a value of the detection and recognition loss function based on the predicted label and a real label of the fundus image sample pair. The category prediction and loss function calculation module specifically includes: calculating a value of a global loss function based on a logarithmic loss between the predicted label of the fundus image sample pair and the real label of the fundus image sample pair; calculating a value of a multi-label loss function based on a sum of a binary cross-entropy loss between the predicted label of the left fundus image and the label of the left fundus image and a binary cross-entropy loss between the predicted label of the right fundus image and the label of the right fundus image; calculating a value of a difference consistency loss function based on a binary cross-entropy loss between a first difference value between the predicted label of the left fundus image and the predicted label of the right fundus image and a second difference value between the label of the left fundus image and the label of the right fundus image; performing weighted summation on the value of the global loss function, the value of the multi-label loss function, and the value of the difference consistency loss function to obtain a value of the detection and recognition loss function; The model training and obtaining module is configured to iteratively train the double-branch feature extraction module, the double-branch attention module, and the classification module until the value of the detection and recognition loss function is minimum, and obtain a trained fundus image detection and recognition model.

9. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the image detection and recognition method according to any one of claims 1-7.

Images