Image detection and recognition method and device and computer readable storage medium

Abstract

The invention belongs to the technical field of image detection and recognition, and relates to an image detection and recognition method and device and a computer readable storage medium. The method comprises the following steps: taking fundus image sample pairs with same labels of left eye bottom images and right eye bottom images in fundus image sample pairs as a group of fundus image sample pairs, and endowing each group of fundus image sample pairs with a new label as a real label; inputting the left and right fundus images of the fundus image sample pair into two feature extraction branches, and outputting a first feature map and a second feature map; inputting the first feature map and the second feature map into two attention branches, and outputting a first depth feature map and a second depth feature map; and splicing the first depth feature map and the second depth feature map, inputting the spliced first depth feature map and the spliced second depth feature map into a classification module, outputting a prediction label of a fundus image sample pair, calculating a value of a detection and recognition loss function, and training a double-branch feature extraction module, a double-branch attention module and the classification module to obtain a trained fundus image detection and recognition model.

Description

Technical Field

[0001] The present invention relates to the field of image detection and recognition technology, and in particular to an image detection and recognition method, device and computer-readable storage medium. Background Art

[0002] The eye is the primary organ for acquiring external information. Abnormalities can lead to visual impairment and impact quality of life. Traditional fundus abnormality testing relies on manual analysis of fundus images to determine the presence and nature of abnormalities. With the continuous advancement of deep learning, an increasing number of neural networks are being used for fundus image analysis and recognition, gradually improving the efficiency and accuracy of fundus abnormality detection and recognition.

[0003] In the prior art, when performing fundus abnormality detection and identification, fundus images are subjected to feature extraction and classification by constructing convolutional neural networks of various simple or complex structures. However, due to the large number of types of fundus abnormalities, fundus images of different abnormalities exhibit different characteristics. At the same time, there may be a fundus image with multiple abnormality categories, which makes the fundus abnormality detection and identification problem more complicated. When faced with fundus images with multiple abnormal features, convolutional neural networks find it difficult to accurately identify various abnormality categories and make correct prediction results. In addition, since fundus abnormalities usually have bilateral correlation characteristics, the prior art only extracts features from unilateral fundus images, ignoring the effective feature information that may exist in the other side of the fundus image, which also leads to low accuracy of fundus abnormality detection and identification results.

[0004] In summary, existing fundus abnormality detection and recognition methods have the problem of low accuracy of detection and recognition results. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low accuracy of detection and identification results in the prior art fundus abnormality detection and identification methods.

[0006] To solve the above technical problems, the present invention provides an image detection and recognition method, comprising: Obtain fundus image sample pairs containing left and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; treat fundus image sample pairs with the same labels for the left and right fundus images as a group of fundus image sample pairs, and assign a new label to each group of fundus image sample pairs as its true label; Inputting the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the dual-branch feature extraction module respectively, and outputting a first feature map of the left fundus image and a second feature map of the right fundus image; Input the first feature map and the second feature map into the two attention branches of the dual-branch attention module respectively, and output a first depth feature map of the left fundus image and a second depth feature map of the right fundus image; The first depth feature map and the second depth feature map are spliced ​​and input into the classification module, the predicted label of the fundus image sample pair is output, and the value of the detection and recognition loss function is calculated based on the predicted label and the true label; The dual-branch feature extraction module, the dual-branch attention module and the classification module are iteratively trained until the value of the detection and recognition loss function is minimized, thereby obtaining a trained fundus image detection and recognition model.

[0007] Preferably, the first depth feature map and the second depth feature map are spliced ​​and input into a classification module, and the predicted label of the fundus image sample pair is output, including: Splicing the first depth feature map and the second depth feature map to obtain a spliced ​​feature map; Use the global average pooling module to perform global spatial information compression on the first depth feature map, the second depth feature map, and the spliced ​​feature map to obtain the first feature vector, the second feature vector, and the global feature vector; The first eigenvector is input into the first fully connected layer, and the predicted label of the left fundus image is output; the second eigenvector is input into the second fully connected layer, and the predicted label of the right fundus image is output; the global eigenvector is input into the third fully connected layer, and the predicted label of the fundus image sample pair is output.

[0008] Preferably, the value of the detection and recognition loss function is calculated based on its predicted label and the true label, including: Calculate the value of the global loss function based on the logarithmic loss between the predicted label and the true label of the fundus image sample pair; The value of the multi-label loss function is obtained based on the binary cross entropy loss between the predicted label of the left fundus image and its label, and the sum of the binary cross entropy loss between the predicted label of the right fundus image and its label; Calculating a first difference between the predicted label of the left fundus image and the predicted label of the right fundus image, calculating a second difference between the label of the left fundus image and the label of the right fundus image, and calculating a value of a difference consistency loss function based on a binary cross entropy loss of the first difference and the second difference; The value of the detection and recognition loss function is obtained by weighted summing the values ​​of the global loss function, the multi-label loss function, and the difference consistency loss function.

[0009] Preferably, the calculation formula of the global loss function is: , in, represents the global loss function; ∈ ; K represents the total number of true labels after label conversion, Predict the probability that the fundus image sample pair belongs to the kth category; is the true probability that the fundus image sample pair belongs to the kth category; The calculation formula of the multi-label loss function is: , in, represents the multi-label loss function; ∈ , N represents the total number of labels before label conversion, ∈ ; represents the predicted probability that the left fundus image belongs to the i-th category; represents the true probability that the left fundus image belongs to the i-th category; represents the predicted probability that the right fundus image belongs to the i-th category; represents the true probability that the right fundus image belongs to the i-th category; The calculation formula of the difference consistency loss function is: , in, represents the difference consistency loss function; , represents the first difference in the predicted probability that the left and right fundus images belong to the i-th class, The second difference value representing the true probability of the left and right fundus images belonging to the i-th class; The calculation formula of the detection and recognition loss function is: , in, Represents the detection and recognition loss function; express The weight parameter of express The weight parameter of express The weight parameter of .

[0010] Preferably, the two attention branches in the dual-branch attention module have different structures, and the first feature map and the second feature map are respectively input into the two attention branches in the dual-branch attention module, comprising: Input the first feature map into the first attention branch structure, and input the second feature map into the second attention branch structure; or, The first feature map is input into the second attention branch structure, and the second feature map is input into the first attention branch structure.

[0011] Preferably, the step of performing deep feature extraction on the input feature map by the first attention branch in the dual-branch attention module comprises: Using multiple first convolution kernels with the same number of channels as the input feature map, convolution is performed on each channel of the input feature map respectively, and multiple feature maps with the same number of channels as the input feature map are output; using a second convolution kernel with the same convolution kernel size as the number of channels of the input feature map, weighted fusion is performed on the multiple feature maps to obtain a first fused feature map; Perform multiple point-by-point convolution operations on the input feature map using multiple series-connected first point-by-point convolution kernels, and obtain a second fused feature map based on the output of the last first point-by-point convolution kernel; The first fused feature map and the second fused feature map are fused element-wise and point-by-point to output a depth feature map.

[0012] Preferably, the step of performing deep feature extraction on the input feature map by the second attention branch in the dual-branch attention module comprises: Perform multiple point-by-point convolution operations on the input feature map using multiple series-connected second point-by-point convolution kernels, and obtain a third fused feature map based on the output of the last second point-by-point convolution kernel. The cross-spatial multi-scale attention module is used to adaptively recalibrate the channel weights of the input feature map and output a global feature map containing the weight information of each channel; The third fused feature map and the global feature map are fused element-wise and point-by-point to output a deep feature map.

[0013] Preferably, before inputting the left fundus image and the right fundus image in the fundus image sample pair into the dual-branch feature extraction module, the method further includes: Obtaining grayscale histograms of the left fundus image and the right fundus image, and if the width of the pixel distribution interval in the grayscale histogram is less than a preset width, and / or the number of pixels in the high grayscale area or the low grayscale area in the grayscale histogram is greater than or equal to a preset number, then using the grayscale histogram as the grayscale histogram to be repaired; The histogram equalization method is used to optimize the grayscale histogram to be repaired, and the optimized fundus image is obtained based on the optimized grayscale histogram to be repaired.

[0014] The present invention also provides an image detection and recognition device, comprising: The data acquisition and label conversion module is used to obtain fundus image sample pairs containing left and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; fundus image sample pairs with the same labels for the left and right fundus images are regarded as a group of fundus image sample pairs, and a new label is assigned to each group of fundus image sample pairs as its true label; a global feature extraction module, configured to input the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the dual-branch feature extraction module, respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image; A depth feature extraction module is configured to input the first feature map and the second feature map into two attention branches in the dual-branch attention module, respectively, and output a first depth feature map of the left fundus image and a second depth feature map of the right fundus image; A category prediction and loss function calculation module is used to splice the first depth feature map and the second depth feature map and input them into the classification module, output the predicted label of the fundus image sample pair, and calculate the value of the detection and recognition loss function based on its predicted label and true label; The model training and acquisition module is used to iteratively train the dual-branch feature extraction module, the dual-branch attention module and the classification module until the value of the detection and recognition loss function is minimized, thereby obtaining a trained fundus image detection and recognition model.

[0015] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image detection and recognition method are implemented.

[0016] The image detection and recognition method provided by this application has the following beneficial effects: 1. First, the multi-classification problem is converted into a single-classification problem, that is, by identifying the normal or abnormal category combination contained in the labels of all fundus image sample pairs in the training set, a new label is created for each group of fundus image sample pairs, so that each fundus image sample pair in the training set corresponds to a label, rather than a label corresponding to the left fundus image and a label corresponding to the right fundus image. This allows the model to focus on learning the feature information of a single category, thereby improving the accuracy of fundus image detection and recognition; further, this application considers that the feature information of the left and right fundus images is correlated, but if the same feature extraction network is used to extract features from the two images, the extracted bilateral features may interfere with each other. Therefore, this application designs A dual-branch feature extraction module and a dual-branch attention module are designed based on a dual-branch architecture to fully extract the feature information of the left and right fundus images, fuse the feature information of the left and right fundus images before classification, and finally directly output the prediction results of abnormality detection and recognition of the fundus image sample pairs. The symmetry and correlation between the bilateral structures of the fundus are taken into account, and the detection and recognition loss is calculated based on the prediction results and the converted labels to train the model. Therefore, the solution provided by this application not only solves the problem of multi-category label confusion in the fundus anomaly detection task that causes the model to be unable to effectively learn single-category features, but also avoids the problem of information loss in unilateral fundus image detection and recognition, thereby improving the accuracy of fundus anomaly detection and recognition. 2. When constructing the detection and recognition loss function, in addition to considering the prediction loss between the predicted label output based on the binocular fusion feature and its true label, the present application also performs classification prediction on the left and right fundus images respectively, and constructs a multi-label independent prediction loss based on the predicted category of the left fundus image and its normal or abnormal category, as well as the predicted category of the right fundus image and its normal or abnormal category; and, considering the consistency and differentiation features between the left and right eyes, constructs a difference consistency loss based on the difference between the predicted labels of the left and right fundus images, and the difference between the normal or abnormal categories of the left and right fundus images; considering both the binocular fusion features and the independent features of the left and right fundus images, it also considers the alignment of the constraint on the predicted difference between the left and right eyes and the true difference, thereby matching the single classification task and the binocular differentiation recognition task during fundus abnormality recognition, and further improving the accuracy of fundus image detection and recognition results; 3. The two attention branches in the dual-branch attention module in this application adopt different structures. The first attention branch structure is used for local feature extraction, which reduces the amount of computation while enhancing the sensitivity to local key structures (such as blood vessels, texture details, etc.); the second attention branch structure is used for global context feature extraction, which captures global context dependencies through cross-dimensional interaction, adaptively highlights channels related to abnormal features (such as deep vascular features related to glaucoma), and suppresses irrelevant channels, establishing semantic consistency associations between left and right eye features; local-global complementarity can be achieved through two attention branches with different structures. Branch one captures the local lesion details of a single eye, and branch two establishes a global consistency association between both eyes. The two are spliced ​​together to form a fusion feature of local detail enhancement + global semantic alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 Flowchart of the image detection and recognition method provided in this application; Figure 2 An image preprocessing flow chart provided for this application; Figure 3 This is a schematic diagram of fundus images before and after background clipping provided in this application; wherein, Figure 3 (a) in the figure is the fundus image before background clipping. Figure 3 (b) is the fundus image after background cropping; Figure 4 This is a schematic diagram of fundus images before and after histogram equalization provided in this application; wherein, Figure 4 (a) in the figure is the fundus image before histogram equalization. Figure 4 (b) in the figure is the fundus image after histogram equalization; Figure 5 Schematic diagram of the label conversion principle provided for this application; Figure 6 Schematic diagram of the image detection and recognition model architecture provided for this application; Figure 7 Schematic diagram of the cross-spatial multi-scale attention module structure provided by this application; Figure 8 This is a schematic diagram of the structure of the image detection and recognition device provided in this application. DETAILED DESCRIPTION

[0018] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0019] See also Figure 1 , Figure 1 The figure shows a flow chart of the image detection and recognition method provided by this application, which specifically includes: S10: Obtain fundus image sample pairs including left fundus images and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; treat fundus image sample pairs with the same labels of the left fundus image and the right fundus image as a group of fundus image sample pairs, and assign a new label to each group of fundus image sample pairs as its true label.

[0020] Specifically, each fundus image sample pair includes a left fundus image and a right fundus image belonging to the same object, and the label of each fundus image sample pair includes the normal or abnormal category of the left fundus image and the normal or abnormal category of the right fundus image; for example, if the left fundus image in a fundus image sample pair is normal, and the right fundus image is simultaneously judged as Class A abnormal and Class B abnormal, then the label of the fundus image sample pair is that the left fundus image is normal and the right fundus image is Class A abnormal + Class B abnormal.

[0021] S20: Input the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the dual-branch feature extraction module respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image.

[0022] S30: Input the first feature map and the second feature map into the two attention branches in the dual-branch attention module respectively, and output a first depth feature map of the left fundus image and a second depth feature map of the right fundus image.

[0023] S40: splicing the first depth feature map and the second depth feature map and inputting them into the classification module, outputting the predicted label of the fundus image sample pair, and calculating the value of the detection and recognition loss function based on its predicted label and the true label.

[0024] S50: Iteratively train the dual-branch feature extraction module, the dual-branch attention module, and the classification module until the value of the detection and recognition loss function is minimized, thereby obtaining a trained fundus image detection and recognition model.

[0025] Since different fundus images have large differences in lighting conditions, image resolution, contrast and color saturation, if there are differences between the fundus image sample pairs in the training set or the left and right fundus images in the fundus image sample pairs, it will lead to inconsistent image recognition and feature extraction accuracy of the detection and recognition model, affecting the accuracy of the fundus image detection and recognition results. Therefore, some embodiments of the present application pre-process the fundus image sample pairs to eliminate these differences.

[0026] like Figure 2 The figure shows an image preprocessing flow chart provided by the present application, which specifically includes background cropping, histogram equalization, filtering denoising and data enhancement.

[0027] Specifically, the fundus image will contain a large area of ​​black background, which is of no substantial help for fundus abnormality detection and identification. It is equivalent to noise in the fundus image, which will reduce the signal-to-noise ratio of the fundus image and increase the difficulty of the model to extract effective information from the fundus image. Therefore, the black background area of ​​the fundus image can be cropped.

[0028] For example, the pixels of the fundus image can be adjusted to 224*224, such as Figure 3 The figure shows the fundus image before and after background clipping; Figure 3 (a) in the figure is the fundus image before background clipping. Figure 3 (b) is the fundus image after background clipping. Figure 3 As can be seen from (a) and (b), the cropped fundus image is more focused on the effective information area.

[0029] Furthermore, due to differences in imaging conditions, some fundus images may show abnormal overall brightness (overexposure or underexposure), and at the same time, there may be similar color between the vascular structure and the optic disc area, which will also affect the accuracy of subsequent feature extraction and recognition. Therefore, in order to optimize the quality of fundus images and improve the recognizability of abnormal fundus features, the contrast of fundus images can be enhanced through histogram equalization.

[0030] Specifically, before step S20, the present application further includes: Obtain the grayscale histograms of the left fundus image and the right fundus image. If the width of the pixel distribution interval in the grayscale histogram is less than the preset width, and / or the number of pixels in the high grayscale area or the low grayscale area in the grayscale histogram is greater than or equal to the preset number, then the grayscale histogram is used as the grayscale histogram to be repaired.

[0031] The histogram equalization method is used to optimize the grayscale histogram to be repaired, and the optimized fundus image is obtained based on the optimized grayscale histogram to be repaired.

[0032] Overexposure of fundus images is manifested as a large number of pixels in the grayscale histogram concentrated in the high grayscale area (pixel value 180-255 area), while underexposure of fundus images is manifested as a large number of pixels in the grayscale histogram concentrated in the low grayscale area (pixel value 0-50 area). Insufficient contrast of fundus images is manifested as the distribution interval of pixels in the grayscale histogram is too narrow, concentrated in a narrow range, which will lead to a weakening of the grayscale differences between fundus tissue structures, forming a low-contrast fundus image. Therefore, this application analyzes the grayscale histogram to reflect the quality of the fundus image, and uses histogram equalization to improve the grayscale histogram of the low-quality fundus image, thereby obtaining a high-quality fundus image through optimized grayscale histogram mapping.

[0033] As a classic nonlinear contrast enhancement algorithm, Histogram Equalization (HE) optimizes the histogram distribution by dynamically remapping the grayscale levels. It can effectively improve low-quality fundus images and remap the original concentrated grayscale values ​​to a wider dynamic range, thereby obtaining an approximately uniform histogram distribution. It can not only correct the grayscale offset caused by abnormal lighting, but also enhance the grayscale difference between the vascular network and the optic disc area, thus establishing a good visual foundation for subsequent feature extraction and recognition. Figure 4 FIG. 1 is a schematic diagram of fundus images before and after histogram equalization provided by the present application; wherein, Figure 4 (a) in the figure is the fundus image before histogram equalization. Figure 4 (b) is the fundus image after histogram equalization. Figure 4 It can be seen from the figure that the vascular patterns of the fundus image after histogram equalization are clearer and the contrast is significantly enhanced.

[0034] Furthermore, since there are many types of fundus abnormalities, fundus abnormality detection is a multi-classification problem with high complexity, the model needs to learn the associations between multiple abnormality categories. When there are multiple categories of fundus abnormalities, the model needs to accurately predict all abnormality categories, which increases the complexity of fundus abnormality recognition to a certain extent, and in turn affects the accuracy of model detection and recognition.

[0035] Based on the above reasons, step S10 of the present application first converts the multi-classification problem into a single-classification problem, that is, by identifying the abnormal category combinations contained in the labels of all fundus image sample pairs in the training set, a new label is created for each group of fundus image sample pairs. For example, if the fundus images in the training set include normal and 7 abnormal categories, there will be 255 category combinations considering only the combinations between categories. However, in actual applications, some categories are mutually exclusive. For example, fundus images cannot be both normal and abnormal. Therefore, the normal category and the remaining abnormal categories cannot appear at the same time. Therefore, the actual category combination is far less than 255. It is worth noting that the present application traverses the public fundus image sample data set and finally obtains 37 different category combinations.

[0036] Specifically, based on the above principle, this application utilizes the mutual exclusivity of fundus image labels to classify all fundus image sample pairs and assign the same new label to each fundus image sample pair in each group of fundus image sample pairs. Figure 5 The figure shows a schematic diagram of the label conversion principle provided by this application. Combining categories and assigning new labels means determining the true label based on the label of each fundus image sample pair. If a fundus image sample pair has both Class C abnormalities and Class O abnormalities, its new label is a label representing the combination of C and O.

[0037] Specifically, the present application uses label conversion to make each fundus image sample pair in the training set correspond to a category, so that the model can focus on learning the feature information of a single category, thereby improving the accuracy of fundus image detection and recognition.

[0038] Furthermore, since the feature information of the left and right fundus images are mutually correlated when performing fundus image detection and recognition, this application considers the timeliness of detection and recognition when constructing the detection and recognition model, and directly uses a dual-branch feature extraction and fusion architecture, such as Figure 6 Shown is a schematic diagram of the detection and recognition model architecture constructed in this application, in which the dual-branch feature extraction module includes two parallel feature extraction branches, which perform global feature extraction on the left fundus image and the right fundus image in a group of fundus image samples respectively. Then, the two attention branches in the dual-branch attention module are used to extract deep features from the left fundus image and the right fundus image, fully tapping their potential complementary information and improving the model's perception of abnormal areas. Finally, the features of the left and right fundus images are fused, and the classification task is completed using the classification model.

[0039] Specifically, the two feature extraction branches in the dual-branch feature extraction module are independent feature extraction networks with the same structure. The feature extraction network can be selected from mainstream convolutional neural network models pre-trained on the ImageNet dataset, such as ResNet, DenseNet, and MobileNet. These models can effectively and automatically learn discriminative feature representations from large-scale fundus images by stacking convolutional layers, pooling layers, and fully connected layers. For example, ResNet alleviates the gradient vanishing problem in deep networks by introducing a residual connection mechanism, making it easier to capture high-level semantic information of the image; DenseNet promotes the transfer and reuse of features through dense connections between layers, improving the expressive power of the model; and MobileNet achieves an effective trade-off between model depth, width, and resolution, using depthwise separable convolution to significantly reduce computational complexity.

[0040] During the research process, this application conducted a comparative evaluation of various feature extraction networks. The results showed that MobileNet_V2 achieved a better balance between accuracy and model complexity. Therefore, as a preferred feature extraction network, the lightweight MobileNet_V2 is used.

[0041] Furthermore, taking into account the characteristics of the dual-branch architecture, we input the features of the left and right eye images into two independent attention modules, named attention1 and attention2, respectively, to enhance their respective responsiveness to key regions and suppress redundant or noisy information. One branch employs an attention mechanism based on depthwise separable convolution (DSConv) and pointwise convolution (PConv). DSConv decomposes the standard convolution into channel-wise and point-wise convolutions, significantly reducing the number of parameters and computational cost while maintaining strong expressive power. The PConv output is then fed through batch normalization (BN) to improve training stability, and the ReLU activation function enhances the network's nonlinear modeling capabilities. The other branch incorporates an efficient multi-scale attention mechanism. This module partitions the channel dimension into multiple subspaces and reconstructs them to the batch dimension, preserving full channel feature information while reducing computational overhead. The EMA module leverages global context to adaptively recalibrate channel weights and enhances pixel-level semantic relationship modeling through cross-dimensional information interaction. Finally, the outputs of the two attention branches are connected in the channel dimension to achieve efficient fusion of left and right eye image information, providing richer feature representation for subsequent classification modules.

[0042] Specifically, the two attention branches in the dual-branch attention module have different structures: the first attention branch structure is used for local feature extraction, which uses DSConv and PConv to reduce the amount of computation while enhancing sensitivity to local key structures (such as blood vessels, texture details, etc.); the second attention branch structure is used for global context feature extraction, which captures global context dependencies through cross-dimensional interaction, adaptively highlights channels related to abnormal features (such as deep vascular features related to glaucoma), and suppresses irrelevant channels, establishing semantic consistency associations between left and right eye features; local-global complementarity can be achieved through two attention branches with different structures: branch one captures the local lesion details of a single eye, and branch two establishes global consistency associations between both eyes. The two are spliced ​​together to form a fusion feature of local detail enhancement + global semantic alignment.

[0043] Furthermore, the first feature map and the second feature map are respectively input into the two attention branches in the dual-branch attention module, including: inputting the first feature map into the first attention branch structure, and inputting the second feature map into the second attention branch structure; or, inputting the first feature map into the second attention branch structure, and inputting the second feature map into the first attention branch structure.

[0044] This application does not limit which attention branch is used to perform deep feature extraction on the first feature map and the second feature map. In actual use, the corresponding attention branch can be selected by pre-analyzing the characteristics of the left and right fundus images.

[0045] Furthermore, the first attention branch in the dual-branch attention module performs the following steps on the input feature map to extract deep features: Using multiple first convolution kernels with the same number of channels as the input feature map, convolution is performed on each channel of the input feature map respectively, and multiple feature maps with the same number of channels as the input feature map are output; using a second convolution kernel with the same convolution kernel size as the number of channels of the input feature map, weighted fusion is performed on the multiple feature maps to obtain a first fused feature map; Perform multiple point-by-point convolution operations on the input feature map using multiple series-connected first point-by-point convolution kernels, and obtain a second fused feature map based on the output of the last first point-by-point convolution kernel; The first fused feature map and the second fused feature map are fused element-wise and point-by-point to output a depth feature map.

[0046] The steps for the second attention branch in the dual-branch attention module to extract deep features from the input feature map include: Perform multiple point-by-point convolution operations on the input feature map using multiple series-connected second point-by-point convolution kernels, and obtain a third fused feature map based on the output of the last second point-by-point convolution kernel. The cross-spatial multi-scale attention module is used to adaptively recalibrate the channel weights of the input feature map and output a global feature map containing the weight information of each channel; The third fused feature map and the global feature map are fused element-wise and point-by-point to output a deep feature map.

[0047] Among them, the cross-space multi-scale attention module is an existing module, such as Figure 7 Shown is a schematic diagram of the structure of the cross-spatial multi-scale attention module.

[0048] Furthermore, when performing detection and identification based on depth feature maps, the present application not only considers the spliced ​​depth feature map that integrates the feature information of the left and right eyes, but also performs detection and identification on the first depth feature map corresponding to the left fundus image and the second depth feature map corresponding to the right fundus image, respectively, so as to further explore the symmetry and difference features between the left and right fundus images and improve the detection and identification accuracy.

[0049] Specifically, the first depth feature map and the second depth feature map are spliced ​​and input into the classification module, and the predicted label of the fundus image sample pair is output, including: Splicing the first depth feature map and the second depth feature map to obtain a spliced ​​feature map; Use the global average pooling module to perform global spatial information compression on the first depth feature map, the second depth feature map, and the spliced ​​feature map to obtain the first feature vector, the second feature vector, and the global feature vector; The first eigenvector is input into the first fully connected layer, and the predicted label of the left fundus image is output; the second eigenvector is input into the second fully connected layer, and the predicted label of the right fundus image is output; the global eigenvector is input into the third fully connected layer, and the predicted label of the fundus image sample pair is output.

[0050] Specifically, the global average pooling module compresses the previously extracted feature maps in the spatial dimension to generate feature vectors with global semantic information. This not only preserves the key features of the fundus image, but also significantly reduces the number of parameters and avoids the risk of overfitting. These feature vectors are then further abstracted and transformed by fully connected layers to meet the requirements of the classification task, outputting the predicted probabilities of the fundus image sample pair belonging to each category, as well as the probabilities of the left and right fundus images belonging to the normal or abnormal category, respectively.

[0051] Furthermore, this application performs label conversion and uses attention branches with different structures to extract differentiated deep features of the left and right fundus images, and then constructs a detection and recognition loss function based on the consistency and difference of the left and right eyes to match single classification tasks and binocular differential detection and recognition tasks.

[0052] Specifically, the value of the detection and recognition loss function is calculated based on its predicted label and the true label, including: Based on the logarithmic loss between the predicted label and the true label of the fundus image sample pair, the value of the global loss function is calculated. The specific calculation formula is: , in, represents the global loss function; ∈ ; K represents the total number of true labels after label conversion, Predict the probability that the fundus image sample pair belongs to the kth category; is the true probability that the fundus image sample pair belongs to the kth category.

[0053] The value of the multi-label loss function is obtained based on the binary cross entropy loss between the predicted label of the left fundus image and its label, and the sum of the binary cross entropy loss between the predicted label of the right fundus image and its label. The specific calculation formula is: , in, represents the multi-label loss function; ∈ , N represents the total number of labels before label conversion, ∈ ; represents the predicted probability that the left fundus image belongs to the i-th category; represents the true probability that the left fundus image belongs to the i-th category; represents the predicted probability that the right fundus image belongs to the i-th category; represents the true probability that the right fundus image belongs to the i-th category.

[0054] Calculate the first difference between the predicted label of the left fundus image and the predicted label of the right fundus image, calculate the second difference between the label of the left fundus image and the label of the right fundus image, and calculate the value of the difference consistency loss function based on the binary cross entropy loss of the first difference and the second difference. The specific calculation formula is: , in, represents the difference consistency loss function; , represents the first difference in the predicted probability that the left and right fundus images belong to the i-th class, The second difference value represents the true probability that the left and right fundus images belong to the i-th class.

[0055] The value of the detection and recognition loss function is obtained by weighted summing the values ​​of the global loss function, the multi-label loss function, and the difference consistency loss function. The specific calculation formula is: , in, Represents the detection and recognition loss function; express The weight parameter of express The weight parameter of express The weight parameter of .

[0056] Based on the image detection and recognition method provided in the above embodiment, the embodiment of the present application also provides an image detection and recognition device, such as Figure 8 As shown, the device specifically includes:

[0057] The data acquisition and label conversion module 10 is used to obtain fundus image sample pairs including left fundus images and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; the fundus image sample pairs with the same labels of the left fundus image and the right fundus image are regarded as a group of fundus image sample pairs, and a new label is assigned to each group of fundus image sample pairs as its true label.

[0058] The global feature extraction module 20 is used to input the left fundus image and the right fundus image in the fundus image sample pair into the two feature extraction branches of the dual-branch feature extraction module respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image.

[0059] The depth feature extraction module 30 is used to input the first feature map and the second feature map into the two attention branches in the dual-branch attention module respectively, and output the first depth feature map of the left fundus image and the second depth feature map of the right fundus image.

[0060] The category prediction and loss function calculation module 40 is used to splice the first depth feature map and the second depth feature map and input them into the classification module, output the predicted label of the fundus image sample pair, and calculate the value of the detection and recognition loss function based on its predicted label and true label.

[0061] The model training and acquisition module 50 is used to iteratively train the dual-branch feature extraction module, the dual-branch attention module and the classification module until the value of the detection and recognition loss function is minimized, thereby obtaining a trained fundus image detection and recognition model.

[0062] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image detection and recognition method are implemented.

[0063] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0065] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0067] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. An image detection and recognition method, characterized in that: include: Obtain fundus image sample pairs including left fundus images and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; The fundus image sample pairs with the same labels for the left fundus image and the right fundus image are regarded as a group of fundus image sample pairs, and a new label is assigned to each group of fundus image sample pairs as its true label; Inputting the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the dual-branch feature extraction module respectively, and outputting a first feature map of the left fundus image and a second feature map of the right fundus image; Input the first feature map and the second feature map into the two attention branches of the dual-branch attention module respectively, and output a first depth feature map of the left fundus image and a second depth feature map of the right fundus image; The first depth feature map and the second depth feature map are spliced ​​and input into the classification module, the predicted label of the fundus image sample pair is output, and the value of the detection and recognition loss function is calculated based on its predicted label and the true label, which specifically includes: Calculate the value of the global loss function based on the logarithmic loss between the predicted label and the true label of the fundus image sample pair; The value of the multi-label loss function is obtained based on the binary cross entropy loss between the predicted label of the left fundus image and its label, and the sum of the binary cross entropy loss between the predicted label of the right fundus image and its label; Calculating a first difference between the predicted label of the left fundus image and the predicted label of the right fundus image, calculating a second difference between the label of the left fundus image and the label of the right fundus image, and calculating a value of a difference consistency loss function based on a binary cross entropy loss of the first difference and the second difference; The value of the global loss function, the value of the multi-label loss function, and the value of the difference consistency loss function are weighted and summed to obtain the value of the detection and recognition loss function; The dual-branch feature extraction module, the dual-branch attention module and the classification module are iteratively trained until the value of the detection and recognition loss function is minimized, thereby obtaining a trained fundus image detection and recognition model.

2. The image detection and recognition method according to claim 1, characterized in that: The first depth feature map and the second depth feature map are spliced ​​and input into the classification module, and the predicted label of the fundus image sample pair is output, including: Splicing the first depth feature map and the second depth feature map to obtain a spliced ​​feature map; Use the global average pooling module to perform global spatial information compression on the first depth feature map, the second depth feature map, and the spliced ​​feature map to obtain the first feature vector, the second feature vector, and the global feature vector; The first eigenvector is input into the first fully connected layer, and the predicted label of the left fundus image is output; the second eigenvector is input into the second fully connected layer, and the predicted label of the right fundus image is output; the global eigenvector is input into the third fully connected layer, and the predicted label of the fundus image sample pair is output.

3. The image detection and recognition method according to claim 1, characterized in that: The calculation formula of the global loss function is: ， in, represents the global loss function; ∈ ; K represents the total number of true labels after label conversion, Predict the probability that the fundus image sample pair belongs to the kth category; is the true probability that the fundus image sample pair belongs to the kth category; The calculation formula of the multi-label loss function is: ， in, represents the multi-label loss function; ∈ , N represents the total number of labels before label conversion, ∈ ; represents the predicted probability that the left fundus image belongs to the i-th category; represents the true probability that the left fundus image belongs to the i-th category; represents the predicted probability that the right fundus image belongs to the i-th category; represents the true probability that the right fundus image belongs to the i-th category; The calculation formula of the difference consistency loss function is: ， in, represents the difference consistency loss function; , represents the first difference in the predicted probability that the left and right fundus images belong to the i-th class, The second difference value representing the true probability of the left and right fundus images belonging to the i-th class; The calculation formula of the detection and recognition loss function is: ， in, Represents the detection and recognition loss function; express The weight parameter of express The weight parameter of express The weight parameter of .

4. The image detection and recognition method according to claim 1, wherein: The two attention branches in the dual-branch attention module have different structures. The first feature map and the second feature map are input into the two attention branches in the dual-branch attention module respectively, including: Input the first feature map into the first attention branch structure, and input the second feature map into the second attention branch structure; or, The first feature map is input into the second attention branch structure, and the second feature map is input into the first attention branch structure.

5. The image detection and recognition method according to claim 1, characterized in that: The first attention branch in the dual-branch attention module performs deep feature extraction on the input feature map in the following steps: Using multiple first convolution kernels with the same number of channels as the input feature map, convolution is performed on each channel of the input feature map respectively, and multiple feature maps with the same number of channels as the input feature map are output; using a second convolution kernel with the same convolution kernel size as the number of channels of the input feature map, weighted fusion is performed on the multiple feature maps to obtain a first fused feature map; Perform multiple point-by-point convolution operations on the input feature map using multiple series-connected first point-by-point convolution kernels, and obtain a second fused feature map based on the output of the last first point-by-point convolution kernel; The first fused feature map and the second fused feature map are fused element-wise and point-by-point to output a depth feature map.

6. The image detection and recognition method according to claim 1, characterized in that: The steps for the second attention branch in the dual-branch attention module to extract deep features from the input feature map include: Perform multiple point-by-point convolution operations on the input feature map using multiple series-connected second point-by-point convolution kernels, and obtain a third fused feature map based on the output of the last second point-by-point convolution kernel. The cross-spatial multi-scale attention module is used to adaptively recalibrate the channel weights of the input feature map and output a global feature map containing the weight information of each channel; The third fused feature map and the global feature map are fused element-wise and point-by-point to output a deep feature map.

7. The image detection and recognition method according to claim 1, characterized in that: Before the left fundus image and the right fundus image in the fundus image sample pair are input into the dual-branch feature extraction module, the following steps are also included: Obtaining grayscale histograms of the left fundus image and the right fundus image, and if the width of the pixel distribution interval in the grayscale histogram is less than a preset width, and / or the number of pixels in the high grayscale area or the low grayscale area in the grayscale histogram is greater than or equal to a preset number, then using the grayscale histogram as the grayscale histogram to be repaired; The histogram equalization method is used to optimize the grayscale histogram to be repaired, and the optimized fundus image is obtained based on the optimized grayscale histogram to be repaired.

8. An image detection and recognition device, characterized in that: include: The data acquisition and label conversion module is used to obtain fundus image sample pairs containing left and right fundus images and their labels in the training set, where the labels are normal or abnormal categories of the fundus images; fundus image sample pairs with the same labels for the left and right fundus images are regarded as a group of fundus image sample pairs, and a new label is assigned to each group of fundus image sample pairs as its true label; a global feature extraction module, configured to input the left fundus image and the right fundus image in the fundus image sample pair into two feature extraction branches of the dual-branch feature extraction module, respectively, and output a first feature map of the left fundus image and a second feature map of the right fundus image; A depth feature extraction module is configured to input the first feature map and the second feature map into two attention branches in the dual-branch attention module, respectively, and output a first depth feature map of the left fundus image and a second depth feature map of the right fundus image; The category prediction and loss function calculation module is used to splice the first depth feature map and the second depth feature map and input them into the classification module, output the predicted label of the fundus image sample pair, and calculate the value of the detection and recognition loss function based on its predicted label and true label, which specifically includes: Calculate the value of the global loss function based on the logarithmic loss between the predicted label and the true label of the fundus image sample pair; The value of the multi-label loss function is obtained based on the binary cross entropy loss between the predicted label of the left fundus image and its label, and the sum of the binary cross entropy loss between the predicted label of the right fundus image and its label; Calculating a first difference between the predicted label of the left fundus image and the predicted label of the right fundus image, calculating a second difference between the label of the left fundus image and the label of the right fundus image, and calculating a value of a difference consistency loss function based on a binary cross entropy loss of the first difference and the second difference; The value of the global loss function, the value of the multi-label loss function, and the value of the difference consistency loss function are weighted and summed to obtain the value of the detection and recognition loss function; The model training and acquisition module is used to iteratively train the dual-branch feature extraction module, the dual-branch attention module and the classification module until the value of the detection and recognition loss function is minimized, thereby obtaining a trained fundus image detection and recognition model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the image detection and recognition method according to any one of claims 1 to 7 are implemented.

Images