Weakly supervised object detection method and related device

Abstract

The application provides a target detection method and related equipment based on weak supervision, a target detector is obtained through weak supervision training by using sample fundus images with image-level class labels, and compared with instance-level label annotation by an ophthalmologist, the method reduces labor cost and annotation error rate, improves target detector training efficiency and accuracy, and inputs a fundus image to be detected into the target detector, extracts a feature map and multiple candidate regions of a target object to be detected, processes the feature map and the multiple candidate regions through a feature mapping network including attention networks, a spatial pyramid pooling network, a feature embedding network and a graph neural network in sequence, obtains target region feature vectors corresponding to the candidate regions, and after processing by a classification network and a regression network, class information and bounding box information of each candidate region are accurately obtained, so that accuracy and reliability of a target detection result of the fundus image are ensured.

Description

Technical Field

[0001] This invention relates primarily to the field of image processing, and more particularly to a target detection method and related equipment based on weak supervision. Background Technology

[0002] Diabetic retinopathy (DR) is a common retinal vascular disease that causes severe vision impairment and is one of the three major complications of diabetes. To achieve rapid and accurate DR screening, deep learning-based classification models are typically used to screen patients' fundus images for DR. Based on the screening results, the DR level can be determined, assisting doctors in quickly and accurately assessing the patient's condition and appropriate treatment.

[0003] Determining the DR (radiational retinal hemorrhage) level typically requires examining the leakage caused by retinal capillary damage to identify the presence of tiny lesions like pinpoint hemorrhages in the fundus. However, deep learning-based classification models lack sensitivity to these tiny lesions and cannot accurately identify pinpoint hemorrhages in fundus images. This reduces the accuracy and reliability of DR screening and grading, and may even interfere with doctors' correct diagnosis and treatment of patients. Summary of the Invention

[0004] To address the above problems, this application proposes the following technical solution:

[0005] On the one hand, this application proposes a weakly supervised target detection method, the method comprising:

[0006] Obtain the fundus image to be examined;

[0007] The fundus image is input into the target detector to extract the feature map of the fundus image and multiple candidate regions for the target object to be detected; the target detector is obtained by weakly supervised training on sample fundus images with image-level category labels;

[0008] The feature map and the plurality of candidate regions are processed by the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions; wherein, the feature mapping network includes an attention network, a spatial pyramid pooling network, a feature embedding network and a graph neural network stacked in sequence.

[0009] The classification and regression networks of the target detector are used to classify and detect each target region feature vector and perform bounding box regression processing to obtain the category information and bounding box information of the corresponding candidate region.

[0010] Based on the category information and bounding box information of each of the multiple candidate regions, the target detection result in the fundus image is obtained.

[0011] Optionally, the step of processing the feature map and the plurality of candidate regions through the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions includes:

[0012] The feature map and the plurality of candidate regions are input into the feature mapping network of the target detector. The attention network is used to enhance the candidate region features of each candidate region in the feature map to obtain a first region feature vector for each of the plurality of candidate regions.

[0013] The candidate region features of each candidate region in the feature map are processed by a spatial pyramid pooling network to obtain a second region feature vector of the same preset length for each of the multiple candidate regions.

[0014] The first region feature vector and the second region feature vector of each of the plurality of candidate regions are input into the feature embedding network, and the dimensionality reduction is performed through the weight matrix of the feature embedding network to obtain the third region feature vector of each of the plurality of candidate regions of the preset length.

[0015] The feature map and the plurality of candidate regions are input into the graph neural network for unsupervised pooling aggregation to obtain the fourth region feature vector of the preset length for each of the plurality of candidate regions;

[0016] The feature vectors of the third region and the fourth region are concatenated to obtain the target region feature vectors of each of the multiple candidate regions.

[0017] Optionally, the step of inputting the first region feature vector and the second region feature vector of each of the plurality of candidate regions into the feature embedding network, and performing dimensionality reduction processing through the weight matrix of the feature embedding network to obtain the third region feature vector of each of the plurality of candidate regions of the preset length includes:

[0018] The first region feature vector and the second region feature vector of each of the multiple candidate regions are input into the feature embedding network. The first region feature vector is processed by two fully connected layers of the first size and an activation function layer in sequence to obtain the fifth region feature vector. The second region feature vector is then processed by two fully connected layers of the first size in sequence to obtain the sixth region feature vector.

[0019] Multiply the feature vectors of the fifth region and the sixth region, and then superimpose the resulting feature vector of the seventh region with the feature vector of the fifth region to obtain the feature vector of the eighth region.

[0020] The feature vector of the eighth region is processed sequentially through a fully connected layer of the first size and a fully connected layer of the second size to obtain a third region feature vector of the preset length corresponding to the candidate region.

[0021] Optionally, the step of extracting the feature map of the fundus image and multiple candidate regions for the target object to be detected includes:

[0022] The feature extraction network of the target detector is used to extract features from the fundus image to obtain the feature map of the fundus image;

[0023] The target detector's region extraction network is used to extract object regions from the fundus image, obtaining multiple candidate regions for the target object to be detected.

[0024] Optionally, the step of extracting object regions from the fundus image using the region extraction network of the target detector to obtain multiple candidate regions for the target object to be detected includes:

[0025] The fundus image is input into the region extraction network of the target detector, and the fundus image is over-segmented using a graph-based image segmentation algorithm to obtain multiple segmented regions of the fundus image;

[0026] Based on the selective search algorithm, the multiple segmented regions are processed to obtain multiple candidate regions for the target object to be detected.

[0027] Optionally, the step of processing the multiple segmented regions according to a selective search algorithm to obtain multiple candidate regions for the target object to be detected includes:

[0028] Based on the selective search algorithm, the multiple segmented regions are processed to obtain multiple recommended regions;

[0029] By identifying overlapping regions among the different recommended regions, and using a non-maximum suppression algorithm to filter the multiple recommended regions, multiple candidate regions for the target object to be detected are obtained.

[0030] Optionally, the feature extraction network is a pre-trained convolutional neural network, which includes the Visual Geometry Group (VGG16) network.

[0031] In the process of obtaining the target detector through weakly supervised training using sample fundus images with image-level category labels, the category prediction loss of the sample category information of each candidate region in the sample fundus image and the regression loss of the sample bounding box information are obtained respectively.

[0032] Based on the category prediction loss and the regression loss of each sample candidate region, the region prediction loss corresponding to the sample candidate region is obtained;

[0033] Based on the prediction loss for each region and the image-level category label of the corresponding sample fundus image, the corresponding cross-entropy loss is obtained;

[0034] The target detector is obtained by training the model by minimizing the cross-entropy loss.

[0035] Optionally, obtaining the target detection result in the fundus image based on the category information and bounding box information of each of the plurality of candidate regions includes:

[0036] Based on the category information and bounding box information of each of the multiple candidate regions, the target bounding box information of the fundus hemorrhage points in the fundus image is obtained;

[0037] Output the fundus image and display the target bounding box of the fundus hemorrhage point on the fundus image.

[0038] On the other hand, this application also proposes a target detection device based on weak supervision, the device comprising:

[0039] The fundus image acquisition module is used to acquire fundus images to be detected;

[0040] An extraction module is used to input the fundus image into a target detector, extract the feature map of the fundus image, and extract multiple candidate regions for the target object to be detected; the target detector is obtained by weakly supervised training of sample fundus images with image-level category labels;

[0041] The target region feature vector acquisition module is used to process the feature map and the plurality of candidate regions through the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions; wherein, the feature mapping network includes an attention network, a spatial pyramid pooling network, a feature embedding network and a graph neural network stacked in sequence;

[0042] The prediction processing module is used to perform classification detection and bounding box regression processing on each feature vector of the target region through the classification network and regression network of the target detector, respectively, to obtain the category information and bounding box information of the corresponding candidate region;

[0043] The target detection result acquisition module is used to obtain the target detection result in the fundus image based on the category information and bounding box information of each of the multiple candidate regions.

[0044] In another aspect, this application also proposes an electronic device, comprising:

[0045] Communication interface;

[0046] The memory is used to store programs that implement the weakly supervised target detection method described above.

[0047] A processor is used to load and execute the program stored in the memory to implement the weakly supervised target detection method described above.

[0048] In another aspect, this application also proposes a computer-readable storage medium having stored thereon at least one set of computer instructions, which is loaded and executed by a processor to implement the target detection method based on weak supervision as described above.

[0049] As can be seen, this application proposes a weakly supervised target detection method and related equipment. In order to accurately and quickly detect small targets such as hemorrhages in fundus images and reduce the cost of manual annotation, it proposes to obtain a target detector by weakly supervising the training of sample fundus images with image-level category labels. Compared with the instance-level labeling by ophthalmologists, this greatly reduces the manual cost and the labeling error rate, and improves the training efficiency and accuracy of the target detector. In this way, the fundus image to be detected is input into the target detector, and its feature map and multiple candidate regions of the target object to be detected are extracted. Then, the feature map and multiple candidate regions are processed by a feature mapping network containing sequentially stacked attention network, spatial pyramid pooling network, feature embedding network and graph neural network to obtain the target region feature vector of the corresponding candidate region. After processing by classification network and regression network, the category information and bounding box information of each candidate region are accurately obtained, ensuring the accuracy and reliability of the target detection results of the fundus image obtained in this way. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating an optional example of the weakly supervised target detection method proposed in this application.

[0052] Figure 2 A schematic diagram of an optional network structure for a target detector suitable for the weakly supervised target detection method proposed in this application;

[0053] Figure 3 A schematic diagram of an optional network structure of the feature mapping network in the target detector applicable to the weakly supervised target detection method proposed in this application;

[0054] Figure 4 This is a flowchart illustrating another optional example of the weakly supervised target detection method proposed in this application.

[0055] Figure 5 This is a flowchart illustrating another optional example of the weakly supervised target detection method proposed in this application.

[0056] Figure 6 A schematic diagram of an optional network structure of the attention network in the feature mapping network of the target detector applicable to the weakly supervised target detection method proposed in this application;

[0057] Figure 7 A schematic diagram of an optional network structure of the spatial pyramid pooling network in the feature mapping network of the target detector applicable to the weakly supervised target detection method proposed in this application;

[0058] Figure 8 A schematic diagram of an optional network structure for the feature embedding network in the feature mapping network of the target detector applicable to the weakly supervised target detection method proposed in this application;

[0059] Figure 9 A schematic diagram of an optional network structure of the Graphsage network in the feature mapping network of the target detector applicable to the weakly supervised target detection method proposed in this application;

[0060] Figure 10 This is a schematic diagram of an optional embodiment of the target detection device based on weak supervision proposed in this application;

[0061] Figure 11 This is a schematic diagram of the hardware structure of an optional example of an electronic device suitable for the weakly supervised target detection method proposed in this application. Detailed Implementation

[0062] To address the technical problems described in the background section, in order to improve the accuracy of deep learning-based classification models in identifying tiny lesions (i.e., micro-targets) such as hemorrhages, a large number of sample images labeled with these micro-targets can be obtained to train and optimize the classification model. However, this requires ophthalmologists to spend a lot of time and effort on manual labeling. Furthermore, since some micro-lesions are very similar to other tissues in the fundus, such as some hemorrhages being very similar to retinal microaneurysms, labeling micro-lesions is very difficult. Sometimes even ophthalmologists find it difficult to make a judgment, making it hard to avoid labeling errors of micro-lesions, which affects the training efficiency and accuracy of the classification model.

[0063] Thus, in scenarios requiring the detection of tiny targets such as hemorrhages with a diameter of only 4-8 pixels from fundus images of 2000 pixels in size, a weakly supervised target detection method with image-level annotation is proposed, which is more costly than the expensive instance-level annotation required for fully supervised target detection. This method eliminates the need for ophthalmologists to spend a lot of time and effort annotating tiny lesions like hemorrhages in a large number of fundus images, greatly saving annotation costs.

[0064] Based on this, during the weakly supervised training of the object detection model, this application only needs to perform image-level annotation on each sample image in the sample dataset, that is, to determine whether each fundus image is a fundus image of a patient with diabetic retinopathy (DR), without needing to accurately identify tiny lesions such as hemorrhages, and is no longer limited to annotation by ophthalmologists. Afterwards, after extracting features and candidate regions where the target object may be located in each fundus image using the object detection algorithm, if a method such as fastRCNN is used to directly perform ROI (Regions of Interests) pooling on the obtained feature map and multiple candidate regions to directly map each candidate region onto the feature map and obtain the features of each candidate region, this does not consider the context of the entire fundus image, the relationships between different candidate regions, and the different sizes of different candidate regions. The features of each candidate region obtained by direct mapping are not complete or comprehensive, which will affect the accuracy of subsequent regression and classification of each candidate region, thereby reducing the accuracy of object detection.

[0065] In response, after obtaining the feature map of the fundus image and multiple candidate regions for the target object (such as hemorrhage point), this application proposes a new region feature mapping method. In this process, this application takes into account the contextual information of the fundus image (to enhance the target features), the size difference of different candidate regions, and the accuracy of expressing feature maps at different scales, so as to obtain the features of different candidate regions more accurately, improve the accuracy of subsequent category recognition and object bounding box detection of each candidate region, and improve the training efficiency and accuracy of the target detection model.

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. For ease of description, only the parts related to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. That is, all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0067] In this invention, flowcharts are used to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0068] Reference Figure 1 The diagram below illustrates an optional example of the weakly supervised target detection method proposed in this application. This method can be applied to electronic devices, such as independent physical servers, server clusters integrating multiple physical servers, or cloud servers with cloud computing capabilities, and / or terminal devices with certain data processing capabilities, such as smart medical devices, robots, and desktop computers. Of course, in some scenarios, the terminal device can also interact with the server to implement the weakly supervised target detection method proposed in this application. Based on this, as... Figure 1 As shown, the target detection method based on weak supervision proposed in this embodiment may include:

[0069] Step S11: Obtain the fundus image to be detected;

[0070] In practical applications, this application utilizes medical devices such as fundus cameras to acquire fundus images of a patient's eyes, obtaining what we refer to as fundus images. These images are used to analyze damage and hemorrhage in the retinal capillaries to determine the presence of diabetic retinopathy (DR). Furthermore, it can identify the presence and extent of hemorrhages at the lesion site, thereby determining the DR grade (i.e., the degree of DR). This assists ophthalmologists in accurately understanding the condition and developing appropriate treatment plans. It should be noted that this application does not limit the method of acquiring fundus images.

[0071] Optionally, if the medical device acquiring fundus images has data processing capabilities, it can be used as an electronic device to execute the weakly supervised target detection method proposed in this application. If the medical device does not have data processing capabilities, the fundus images of a particular eye can be transmitted to an electronic device (such as a terminal or server) via wired or wireless communication, so that the electronic device can execute the weakly supervised target detection method proposed in this application on each received fundus image to obtain the target detection result. Of course, the implementation methods of step S11 include, but are not limited to, these two methods, and can be determined according to the application environment. This application will not provide detailed examples of each implementation method.

[0072] Step S12: Input the fundus image into the target detector to extract the feature map of the fundus image and multiple candidate regions for the target object to be detected; the target detector is obtained by weakly supervised training of sample fundus images with image-level category labels;

[0073] Based on the above description of the technical solution of this application, it is unnecessary for ophthalmologists to annotate a large number of fundus images with instance-level labels, such as whether retinal hemorrhages exist and their location information. Only image-level labeling of the fundus images is required to determine whether DR exists. This enables weakly supervised training of the target detector, greatly saving the various costs associated with having ophthalmologists annotate sample data and improving model training efficiency. The training process of the target detector can be referred to the corresponding description in the following embodiments, which will not be detailed here.

[0074] Thus, after obtaining any fundus image, in order to accurately determine whether it is a DR image, whether there are hemorrhages in the retina, and to determine the bounding box information of the retinal hemorrhages, the fundus image can be input into a target detector for processing. During this processing, reference is made to... Figure 2The network structure of the target detector shown can, on the one hand, extract features from the fundus image through a feature extraction network to obtain the corresponding feature map, and on the other hand, extract regions in the fundus image that may be the target object to be detected through a region extraction network to obtain multiple candidate regions in the fundus image. This application does not describe the image feature extraction process and the object region extraction process in detail.

[0075] Step S13: The feature map and multiple candidate regions are processed through the feature mapping network of the target detector to obtain the target region feature vectors of each candidate region; wherein, the feature mapping network includes an attention network, a spatial pyramid pooling network, a feature embedding network and a graph neural network stacked in sequence.

[0076] Based on the above description of the technical solution of this application, the target detector trained in this application maps each candidate region to a feature map. In the process of determining the regional features of each candidate region, such as... Figure 3 The schematic diagram of the feature mapping network structure shown will utilize a feature mapping network composed of sequentially stacked attention networks, spatial pyramid pooling networks, feature embedding networks, and graph neural networks to achieve the normalization processing of feature maps and multiple candidate regions in fundus images. This will obtain the regional feature vectors of each candidate region more accurately, comprehensively, and with fine granularity, denoted as the target region feature vector, thereby improving the accuracy and efficiency of subsequent object category prediction and object bounding box detection based on this.

[0077] The Attention network, based on the attention mechanism, can concentrate limited attention (or information processing resources) on key information (such as the target object to be detected in this application) after determining the target object to be focused on in this application. It uses fewer parameters, saves processing resources, and can quickly obtain the most effective information. This application does not restrict the network structure of the attention network, and it can be determined according to the attention type of the attention mechanism used.

[0078] Spatial Pyramid Pooling (SPP) networks support a highly efficient multi-resolution strategy, compatible with multi-scale features. They can process inputs of arbitrary size into fixed-size outputs, thereby obtaining feature vectors of the same size for candidate regions of different sizes. This avoids the drawbacks of repeatedly extracting features from each candidate region, thus avoiding the pitfalls of processing input images into a fixed size. Furthermore, by extracting spatial feature information of different sizes, the model's robustness to spatial layout and target object deformation is improved. This application does not impose restrictions on the network structure of the SPP layer in the feature mapping network; it can be determined as appropriate.

[0079] Feature embedding networks can transform (reduce dimensionality) input data into fixed-size feature representations (vectors) to meet the input processing requirements of subsequent networks. This dimensionality reduction method can be compared to a fully connected layer (which has no activation function). Through the weight matrix of the feature embedding layer, the feature dimension is reduced to obtain a fixed-size region feature vector.

[0080] Graph neural networks (Graph Simple and AggregateGatE) are an inductive learning framework that efficiently generates embeddings for unknown vertices using vertex attribute information. They generate embedding vectors for target vertices by learning a function that aggregates the representations of neighboring vertices. Simply put, they randomly select N nodes (with replacement) from the current node's neighbors and aggregate the features of these N nodes to generate the representation of the current node. This application utilizes this unsupervised pooling aggregation method, which effectively captures different aspects of neighborhood features, is more conducive to representing nodes, and improves the accuracy of the output region feature vector. This application does not elaborate on the network structure and working principle of graph neural networks.

[0081] Based on the above analysis, this application uses a feature mapping network with the above structure to replace the last pooling layer (pool5) of the activation function layer (i.e., ReLU layer, such as relu5) in the last convolutional block of the convolutional neural network (feature extraction network), so as to obtain the regional feature vectors of each candidate region of the target object to be detected more accurately, thereby improving the accuracy and completeness of the object feature representation of the regional feature vector.

[0082] Step S14: The classification and regression networks of the target detector are used to classify and detect the feature vector of each target region and perform bounding box regression processing to obtain the category information and bounding box information of the corresponding candidate region.

[0083] Following the above analysis, after obtaining the target region feature vectors of multiple candidate regions of the target object to be detected in the fundus image, the object category of each candidate region can be further predicted based on these vectors. That is, the prediction probability of each candidate box belonging to a different type is determined, and the category corresponding to the highest prediction probability is determined as the category information of the corresponding candidate region. At the same time, using the target region feature vectors, regression processing is performed on the corresponding candidate regions to accurately predict the target detection box of the target object, that is, the bounding box information of the target object in the corresponding candidate region. This application does not describe in detail the network structure and processing principle of the classification network and the regression network.

[0084] Step S15: Based on the category information and bounding box information of each of the multiple candidate regions, obtain the target detection results in the fundus image.

[0085] In this embodiment, after weakly supervising the training of sample fundus images with image-level category labels to obtain the target detector, the fundus image of the target object to be detected (which can be a small object in the fundus image, such as a fundus hemorrhage) is input into the target detector to obtain its feature map and multiple candidate regions of the target object. Then, according to the method described above, the feature map and multiple candidate regions are normalized through attention mechanism, spatial pyramid pooling, feature embedding and graph neural network to accurately obtain the target region feature vector of each candidate region of the target object. Compared with the region features obtained by directly mapping the obtained candidate regions to the feature map, the target region feature vectors output by the feature mapping network of this application more accurately express the features of the target object, thereby improving the accuracy and reliability of the subsequent category prediction and bounding box regression.

[0086] Subsequently, based on the category and bounding box information corresponding to each candidate region, the bounding box information belonging to the target object category is determined. This bounding box information can be annotated into the fundus image for output. For example, the bounding box of the detected fundus hemorrhage point (i.e., the target detection box) can be displayed in the output fundus image to assist doctors in accurately determining the DR level of the patient's fundus and formulating a correct and reasonable treatment plan. This application does not limit the representation method and content of the target detection results in the fundus image, including but not limited to the display method described in this embodiment. A detection report in a preset format can also be generated, depending on the situation.

[0087] Reference Figure 4 This is a flowchart illustrating another optional example of the weakly supervised target detection method proposed in this application. This embodiment describes an optional implementation of the weakly supervised training method for the target detector in the weakly supervised target detection method proposed above, such as... Figure 4 As shown, the method may include:

[0088] Step S41: Obtain multiple sample fundus images with image-level category labels;

[0089] The weakly supervised training of the target detector proposed in this application can perform image-level annotation on open-source retinal fundus image datasets. That is, after obtaining multiple sample fundus images from the retinal fundus image dataset, ordinary ophthalmologists only need to identify whether DR exists in each sample fundus image to obtain the corresponding image-level category label. There is no need to accurately identify the bounding boxes of fundus hemorrhage points in the sample fundus images, which reduces the professional skill requirements of ophthalmologists and reduces the manual cost of label annotation.

[0090] Step S42: Input each sample fundus image into the initially constructed target detection network, and process it through the feature extraction network and region extraction network to obtain the sample feature map of the sample fundus image and multiple sample candidate regions of the sample object.

[0091] In practical applications of this application, the feature extraction network can be a pre-trained convolutional neural network, which includes, but is not limited to, the Visual Geometry Group (VGG16) network. This application does not impose restrictions on the network structure of the feature extraction network. This application utilizes the feature extraction capabilities of convolutional neural networks to extract image features. The region extraction network can be constructed using a region recommendation algorithm. This application does not impose restrictions on the implementation method for extracting multiple candidate regions where the target object to be detected may be located from the fundus image.

[0092] Step S43: Input the sample feature map and multiple sample candidate regions into the feature mapping network of the target detection network for processing to obtain the sample region feature vectors corresponding to each of the multiple sample candidate regions.

[0093] Step S44: Input the feature vectors of each sample region into the classification network and regression network of the target detection network respectively, and output the sample category information and sample bounding box information of the corresponding candidate region.

[0094] The implementation process of steps S43 and S44 can be implemented according to the methods described in the corresponding parts of the context embodiment, such as the descriptions of the corresponding parts of steps S13 and S14 above. This embodiment does not describe in detail the implementation process of region feature mapping, classification prediction and bounding box recognition.

[0095] In this embodiment of the application, for ease of description, the outputs of the classification network and the regression network can each be denoted as X. c X d That is, two matrices of dimension C×|R|, i.e. C can represent the number of categories contained in the sample fundus image dataset, and R can represent the number of candidate regions. The values ​​of C and R can be determined as needed.

[0096] Step S45: Obtain the category prediction loss for each sample category information and the regression loss for each sample bounding box information;

[0097] Following the above analysis, for the sample category information Xc output by the classification network, softmax activation can be applied to the C dimension to obtain the corresponding category prediction loss. That is, the probability of each category is calculated based on the softmax activation function, thereby predicting the probability that each candidate box proposal belongs to each category:

[0098]

[0099] In the above formula (1), i and j can represent the number of rows and columns in the output matrix, respectively: x ij This can represent the class prediction result in the i-th row and j-th column of the output matrix, where e represents the logarithm in the softmax activation function; [σ class (X c )] ij This can represent the category prediction loss for the candidate box proposal in the i-th row and j-th column.

[0100] Similarly, the matrix X output by the regression network can be... d Apply softmax activation to the R dimension as shown in formula (2) to predict the contribution of each candidate box proposal to the classification of a specific category (such as the category to which the target object belongs):

[0101]

[0102] In the above formula, [σ det (X d )] ij This can represent the regression loss of the candidate box proposal in the i-th row and j-th column.

[0103] Step S46: Obtain the region prediction loss based on the category prediction loss and regression loss of each candidate region;

[0104] Following the method described above, the category prediction loss and regression loss corresponding to each candidate region of the sample are obtained, i.e., the sample category prediction score σ. class (X c ) and the sample bounding box prediction score σ det (X d The Hadamard multiplication (i.e., element-wise multiplication) of these two rating matrices can be used to obtain the region prediction loss X for the corresponding candidate region. R ,Right now However, it is not limited to this method of obtaining regional prediction loss.

[0105] Step S47: Based on the region prediction loss and the image-level classification label of the corresponding sample fundus image, obtain the cross-entropy loss of the sample fundus image;

[0106] Step S48: The model is trained by minimizing the cross-entropy loss to obtain a target detector for detecting target objects in fundus images; the target objects include fundus hemorrhage points.

[0107] Following the above analysis, the regional prediction loss matrix X... RThe R-dimensional values ​​are summed to obtain the classification prediction score y of the corresponding input sample fundus image. C ,Right now r can represent the target object category. As analyzed above, R is the number of candidate boxes, and C represents the number of categories. Then, the image category represented by the classification prediction score can be compared with the image-level classification label of the corresponding sample fundus image using the cross-entropy loss function. This yields the cross-entropy loss of the image category prediction for the corresponding sample fundus image. By minimizing this cross-entropy loss, the model is trained, resulting in a target detector for detecting target objects in fundus images.

[0108] Cross-entropy loss measures the distance between two probability distributions: the distribution of image-level category labels for each sample fundus image and the probability distribution of the object detector's category predictions for the sample fundus images. The closer these two probability distributions are, the smaller the cross-entropy loss and the higher the accuracy of the model's prediction results. Based on this, the model parameters can be optimized and adjusted using algorithms such as gradient descent to minimize the cross-entropy loss. The implementation process for each training iteration is similar to the steps described above, and will not be detailed in this application.

[0109] It should be understood that, in order to verify the detection accuracy of the trained object detection model, the above-mentioned sample fundus images can be divided into a training dataset and a test dataset. The model is trained using sample fundus images in the training dataset, and the trained object detector is tested using fundus images contained in the test dataset. The testing process is similar to the training process described above. The model parameters of the object detector can be optimized based on the test results. The implementation process is not detailed in this application.

[0110] Reference Figure 5 This is a flowchart illustrating another optional example of the weakly supervised target detection method proposed in this application. This embodiment can describe an optional refined implementation process of the weakly supervised target detection method proposed above, such as... Figure 5 As shown, the method may include:

[0111] Step S51: Obtain the fundus image to be detected;

[0112] Step S52: Input the fundus image into the feature extraction network of the target detector for feature extraction to obtain the feature map of the fundus image;

[0113] Step S53: Input the fundus image into the region extraction network of the target detector, and perform over-segmentation on the fundus image using a graph-based image segmentation algorithm to obtain multiple segmented regions of the fundus image;

[0114] Step S54: Based on the selective search algorithm, process multiple segmented regions to obtain multiple candidate regions for the target object to be detected;

[0115] Based on the description of the corresponding parts of the above embodiments, this application can extract features from fundus images through the feature extraction network of the target detector to obtain feature maps of fundus images, and extract object regions from fundus images through the region extraction network of the target detector to obtain multiple candidate regions for the target object to be detected. This application does not limit the network structure of the feature extraction network and the region extraction network.

[0116] Optionally, this application may employ a VGG16 convolutional neural network to pre-train a feature extractor as the feature extraction network for the object detector, but is not limited to this type of convolutional neural network. For the region extraction network, a Selective Search (SS) algorithm can be used to obtain multiple region proposals in the fundus image. Before executing the SS algorithm, the fundus image can be over-segmented using graph-based image segmentation algorithms such as the Felzenszwalb and Huttenlocher algorithm, resulting in multiple segmented regions, i.e., an over-segmented image. The implementation process is not detailed in this application. Then, the SS algorithm is executed on these segments to obtain multiple candidate regions.

[0117] Based on this, a region set R = {r1, r2, ..., r} of the fundus image is initialized using a graph-based image segmentation algorithm. n In other words, after obtaining multiple segmented regions, superpixels can be merged to merge adjacent regions that are similar to each other (which may include but are not limited to similarity in different attributes such as color, texture, size, and shape), thereby proposing regions from smaller subdivisions to larger subdivisions and obtaining multiple candidate regions for the target object.

[0118] Therefore, during the execution of the SS algorithm, the similarity between adjacent regions in the region set R can be calculated to construct a similarity set S. Then, from this set S, the two regions r with the highest similarity can be determined. i and r j Merge them into a new region r t , will set S with r i and r j Relevant values ​​are removed, and the new region r is calculated. t The similarity to its neighboring regions is calculated, and the results are added to the similarity set S, while the new region r is... tAdding a region to a region set R allows each region in the region set R to be used as the location bounding box L of the target object, i.e., a candidate region.

[0119] In the process of calculating the similarity between the adjacent regions mentioned above, region r can be calculated. i and r j Color similarity S between 颜色 (r i ,r j Texture similarity S 纹理 (r i ,r j Size similarity S 尺寸 (r i ,r j ), and shape similarity S 形状 (r i ,r j This application does not restrict these similarity calculation methods and allows them to be determined as appropriate. Among them, color similarity calculation can merge two adjacent regions with completely identical colors; texture similarity calculation can also merge two adjacent regions with identical textures; size similarity calculation can encourage the priority merging of smaller adjacent regions; shape similarity, i.e., shape compatibility calculation, can measure the degree of cooperation between two regions, so that only adjacent regions can be merged, and two distant regions cannot be merged.

[0120] In this way, this application can combine the above-mentioned similarities to obtain the corresponding two regions r. i and r j The similarity between S(r) i ,r j ),Right now:

[0121] S(r i ,r j )=α1×S 颜色 (r i ,r j )+α2×S 纹理 (r i ,r j )+α3×S 尺寸 (r i ,r j )+α4×S 形状 (r i ,r j );

[0122] In this similarity calculation formula, α1, α2, α3, and α4 represent the weights of the corresponding dimensions of similarity. This application does not impose restrictions on the values ​​of each weight; they can be determined based on the requirements of the actual scenario for each dimension of regional similarity. They can be 0 or 1, corresponding to whether to adopt the similarity of a certain dimension. The similarity values ​​of each dimension are values ​​between [0,1]. Therefore, it can be seen that in different scenarios, and even for different two regions, the similarity dimensions referenced when calculating their similarity can be different, including but not limited to one or more dimensions listed above. This application will not provide detailed examples of each dimension.

[0123] In some embodiments, there may be a lot of overlap between the multiple candidate bounding boxes obtained by the SS algorithm described above. In order to improve the accuracy of the candidate regions, multiple regions obtained by processing multiple segmented regions according to the selective search algorithm can be determined as recommended regions. Then, if it is determined that there are overlapping regions between different recommended regions, the multiple recommended regions can be filtered according to the non-maximum suppression (NMS) algorithm until the candidate boxes with almost no overlap are finally retained, so that almost each target object corresponds to a candidate box, which are determined as multiple candidate regions (usually 2000) for the target object to be detected. The implementation process of further filtering of recommended regions in this application is not described in detail.

[0124] Step S55: Input the feature map and multiple candidate regions into the feature mapping network of the target detector, and enhance the candidate region features in the feature map of each candidate region through the attention network to obtain the first region feature vector for each of the multiple candidate regions.

[0125] In determining the regional characteristics of each candidate region, this application introduces an attention mechanism to quickly obtain important information about the target object of interest using fewer parameters. Figure 6 The diagram shows an optional network structure for an attention network. This attention network may include two fully connected layers of size 1024 connected in sequence, a transpose layer, an activation function layer (such as a softmax layer, which performs numerical normalization to obtain the attention weights), and a fully connected layer of size 512 (denoted as a preset length). The input of the 512-size fully connected layer is obtained by performing a cross product operation on the outputs of the activation function layer and the first 1024-size fully connected layer.

[0126] It should be noted that this application can determine the network structure based on the working principle of the attention mechanism, including but not limited to... Figure 6The network structure shown is not limited to the dimensions of each fully connected layer as described in this embodiment. In practical applications, this application can input the extracted feature maps of the fundus image and multiple candidate regions into the attention network for processing to obtain the first region feature vector of the corresponding candidate region. This processing can be determined according to the type of each network layer constituting the attention network, which will not be described in detail in this application.

[0127] Step S56: The candidate region features in the feature map of each candidate region are processed by the spatial pyramid pooling network to obtain the second region feature vector of the same preset length for each of the multiple candidate regions.

[0128] Spatial Pyramid Pooling (SPP) can extract features using multiple pooling layers of different scales, fuse them into a 21-dimensional vector, and then input it into a fully connected layer of size 512 to obtain a region feature vector of a preset length. Based on this, this application can input the extracted feature map of the fundus image and multiple candidate regions into the SPP network for processing to obtain the second region feature vector of the corresponding candidate region. This application does not describe in detail the processing of the input content by SPP.

[0129] Optional, refer to Figure 7 The diagram shows an optional network structure of the Spatial Pyramid Pooling (SPP) network. For feature extraction networks, features are extracted from fundus images, such as through convolutional layers of a convolutional neural network. The resulting 256-channel feature map can be input into this SPP network. Assuming the size of this feature map is w×h, ... Figure 7 As shown, SPP can divide each candidate region into three sub-images of size 4×4, 2×2, and 1×1. Max pooling is performed on each sub-image to obtain corresponding feature vectors of size 16×256, 4×256, and 1×256. After stacking, a feature vector of size 21×256 for the corresponding candidate region is obtained. This ensures that no matter how large the input fundus image or candidate region is, the SPP network can obtain feature vectors of the same scale. Then, a fully connected layer of size 512 is input to obtain the second region feature vectors of the same preset length for each of the multiple candidate regions.

[0130] As can be seen, this application uses the SPP network to extract features from candidate regions, which solves the problem that convolutional neural networks require input images to have the same size. It can process input images with arbitrary aspect ratios and sizes, as well as candidate regions of different sizes extracted from them, thus improving the accuracy of region feature vectors.

[0131] Step S57: Input the first region feature vector and the second region feature vector of each of the multiple candidate regions into the feature embedding network, and perform dimensionality reduction processing through the weight matrix of the feature embedding network to obtain the third region feature vector of each of the multiple candidate regions with a preset length.

[0132] Based on the above description of the Feature Embedding network, this network can perform dimensionality reduction on features, achieving dimensionality reduction of the region feature vectors output by the attention network and the SPP network respectively.

[0133] Optional, refer to Figure 8 The diagram illustrates an optional network structure for a feature embedding network. After inputting the first and second region feature vectors of multiple candidate regions into the network, the first region feature vector is processed sequentially through two fully connected layers of a first size (e.g., 1024) and an activation function layer (e.g., a sigmoid layer) to obtain a fourth region feature vector. Simultaneously, the second region feature vector is processed sequentially through two fully connected layers of the first size to obtain a fifth region feature vector. Then, the fourth and fifth region feature vectors are multiplied (i.e., a cross product operation), and the resulting sixth and fifth region feature vectors are superimposed to obtain a seventh region feature vector. Finally, the seventh region feature vector is processed sequentially through a fully connected layer of the first size and a fully connected layer of the second size (e.g., 512) to obtain a third region feature vector of a preset length corresponding to the candidate region.

[0134] Step S58: The feature map of the fundus image and the input map neural network of multiple candidate regions are subjected to unsupervised pooling aggregation to obtain the fourth region feature vector of each of the multiple candidate regions with a preset length.

[0135] In this embodiment, the Graphsage algorithm incorporates neighbor sampling and feature aggregation principles. Specifically, it uses connection information between nodes to sample neighbors, and then continuously fuses the information of adjacent nodes through multiple aggregation functions to predict node labels. Based on this, this application proposes an unsupervised improved Graphsage network as the aforementioned graph neural network to achieve node embedding.

[0136] Reference Figure 9The diagram illustrates an optional network structure of the Graphsage network. After inputting the feature map of the fundus image and multiple candidate regions into this network, the feature vectors constituting the feature map can be used as features for different nodes. Starting from the target node (i.e., the node corresponding to the feature vector of the target object), neighboring nodes are searched according to the target depth. The neighboring node information of each node is aggregated layer by layer from the deepest layer upwards until the target node is reached. The final node feature representation is obtained in this way. Based on the aggregation function, the information represented by the child nodes of each layer is aggregated to obtain a better feature representation of the corresponding node, thereby improving the accuracy of the regional feature vectors of each candidate region for subsequent task processing.

[0137] To obtain a fourth region feature vector of the same preset length, a fully connected layer of size 512 can be deployed at the last layer of the graph neural network. The region feature vectors of each candidate region output by Graphsage are input into this fully connected layer for processing to obtain the fourth region feature vector of the preset length. It is evident that this application, through the Graphsage graph neural network, can perform unsupervised pooling aggregation on the input feature map and multiple candidate regions, effectively capturing different dimensional features of adjacent candidate regions, and more comprehensively and accurately expressing the feature representation of the candidate region, i.e., the fourth region feature vector, which helps improve the accuracy of subsequent object category prediction and bounding box detection.

[0138] Step S59: The feature vectors of the third region and the fourth region are concatenated to obtain the target region feature vectors of each of the multiple candidate regions.

[0139] Combination Figure 3 The feature mapping network structure shown, through the processing of feature maps and multiple candidate regions of the input fundus image by its constituent networks, outputs feature vectors for the third and fourth regions. This can be achieved through a concatenation layer (such as...). Figure 3 (The circled C in the text is consistent with the above). The feature vectors of the third and fourth regions of the same candidate region are concatenated and combined to obtain the target region feature vector of the corresponding candidate region. The implementation process is not detailed in this application.

[0140] Step S510: Input the feature vectors of each target region into the classification network and regression network of the target detector respectively to obtain the category information and bounding box information of the corresponding candidate regions.

[0141] The implementation process of step S510 can be referred to the description of the corresponding part of the above embodiment, and will not be described in detail here.

[0142] Step S511: Based on the category information and bounding box information of each of the multiple candidate regions, obtain the target bounding box information of the fundus hemorrhage points in the fundus image;

[0143] Step S512: Output the fundus image and display the target bounding box of the fundus hemorrhage point on the fundus image.

[0144] In this embodiment of the application, the scenario of a hemorrhage point in the retinal blood vessels of the fundus (hereinafter referred to as a fundus hemorrhage point) as the target object is described. For such a small target, the target detector of the above structure can quickly and accurately detect the bounding box of the fundus hemorrhage point in the fundus image, that is, the target detection box, which can be displayed in the fundus image for ophthalmologists to view, assist ophthalmologists in determining whether the patient has DR, and accurately determine the DR level, so as to formulate an appropriate treatment plan.

[0145] Based on the weakly supervised target detection methods described in the above embodiments, in order to accurately detect hemorrhage points in fundus images, this application can utilize sample fundus images labeled with image-level tags to train a target detector through weak supervision. This target detector employs a feature mapping network composed of sequentially stacked attention networks, spatial pyramid pooling networks, feature embedding networks, and graph neural networks to process the feature maps and multiple candidate regions extracted from the sample fundus images. This ensures that the resulting region feature vectors can accurately and comprehensively represent the target object features in candidate regions of any size, improving the accuracy and reliability of subsequent object category detection and bounding box regression, thereby enhancing the training efficiency and detection accuracy of the target detector. The training process of this target detector can be referred to the corresponding descriptions in the above embodiments; it will not be detailed here.

[0146] It should be understood that after each training iteration, this application obtains the image category prediction loss according to the method described above. When adjusting the model parameters of the target detector, the network parameters of each network constituting the target detector can be adjusted to optimize the performance of the target detector. The implementation method of adjusting the model parameters in this application will not be detailed. For the trained target detector, it can be validated / tested using fundus images contained in the validation dataset and test dataset. After confirming that the validation / test results meet the detection requirements, the target detector can be used to implement hemorrhage detection in fundus images.

[0147] Therefore, in the scenario of detecting retinal hemorrhages, based on the descriptions of the above embodiments, a weakly supervised hemorrhage detection method can be adopted, that is, an end-to-end hemorrhage detection method that no longer relies on instance-level annotation, such as annotating the location of retinal hemorrhages, which greatly saves the cost of manual annotation. Moreover, the image-level annotation used in this application greatly reduces the probability of annotation errors compared to instance-level annotation, ensuring the training efficiency and reliability of the target detector.

[0148] Furthermore, based on the description of the feature mapping network above, this application can achieve regional feature extraction of candidate regions of any size, better meeting the needs of small target recognition in computer vision. Thus, in the scenario of detecting retinal hemorrhages with a diameter of approximately 4-8 pixels in a 2000-pixel fundus image, the weakly supervised target detection method proposed in this application can quickly and accurately detect retinal hemorrhages in fundus images of any size. Subsequently, the DR level can be directly determined based on this. For example, if retinal hemorrhages are found in the fundus image, it can be determined as moderate non-proliferative DR, etc., which can be determined according to the relevant DR classification rules of medical institutions, and can assist ophthalmologists in accurately formulating appropriate treatment plans.

[0149] Reference Figure 10 This is a schematic diagram of an optional embodiment of the target detection device based on weak supervision proposed in this application, as shown below. Figure 10 As shown, the device may include:

[0150] Fundus image acquisition module 101 is used to acquire fundus images to be detected;

[0151] The extraction module 102 is used to input the fundus image into the target detector, extract the feature map of the fundus image, and extract multiple candidate regions for the target object to be detected; the target detector is obtained by weakly supervised training of sample fundus images with image-level category labels;

[0152] The target region feature vector acquisition module 103 is used to process the feature map and the plurality of candidate regions through the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions; wherein, the feature mapping network includes an attention network, a spatial pyramid pooling network, a feature embedding network and a graph neural network stacked in sequence.

[0153] The prediction processing module 104 is used to perform classification detection and bounding box regression processing on each feature vector of the target region through the classification network and regression network of the target detector, respectively, to obtain the category information and bounding box information of the corresponding candidate region;

[0154] The target detection result acquisition module 105 is used to obtain the target detection result in the fundus image based on the category information and bounding box information of each of the multiple candidate regions.

[0155] In some embodiments, the target region feature vector acquisition module 103 may include:

[0156] The first region feature vector obtaining unit is used to input the feature map and the plurality of candidate regions into the feature mapping network of the target detector, and to enhance the candidate region features of each candidate region in the feature map through the attention network to obtain the first region feature vector for each of the plurality of candidate regions.

[0157] The second region feature vector acquisition unit is used to process the candidate region features of each candidate region in the feature map through a spatial pyramid pooling network to obtain a second region feature vector of the same preset length for each of the multiple candidate regions.

[0158] The third region feature vector obtaining unit is used to input the first region feature vector and the second region feature vector of each of the plurality of candidate regions into the feature embedding network, and perform dimensionality reduction processing through the weight matrix of the feature embedding network to obtain the third region feature vector of each of the plurality of candidate regions of the preset length.

[0159] The fourth region feature vector acquisition unit is used to input the feature map and the plurality of candidate regions into the graph neural network for unsupervised pooling aggregation processing to obtain the fourth region feature vector of the preset length for each of the plurality of candidate regions;

[0160] The target region feature vector acquisition unit is used to concatenate the feature vector of the third region and the feature vector of the fourth region to obtain the target region feature vector of each of the multiple candidate regions.

[0161] Optionally, the unit obtained from the feature vector of the third region may include:

[0162] The first processing unit is used to input the first region feature vector and the second region feature vector of each of the plurality of candidate regions into the feature embedding network, and process the first region feature vector through two fully connected layers of the first size and an activation function layer in sequence to obtain a fourth region feature vector, and process the second region feature vector through two fully connected layers of the first size in sequence to obtain a fifth region feature vector.

[0163] The second processing unit is used to multiply the feature vector of the fourth region and the feature vector of the fifth region, and then superimpose the resulting feature vector of the sixth region with the feature vector of the fifth region to obtain the feature vector of the seventh region.

[0164] The third processing unit is used to process the feature vector of the seventh region sequentially through the fully connected layer of the first size and the fully connected layer of the second size to obtain the feature vector of the third region of the preset length corresponding to the candidate region.

[0165] In some other embodiments, the extraction module 102 described above may include:

[0166] The feature map obtaining unit is used to extract features from the fundus image through the feature extraction network of the target detector to obtain the feature map of the fundus image;

[0167] The candidate region acquisition unit is used to extract object regions from the fundus image through the region extraction network of the target detector to obtain multiple candidate regions for the target object to be detected.

[0168] Optionally, the candidate region acquisition unit may include:

[0169] The segmented region acquisition unit is used to input the fundus image into the region extraction network of the target detector, and to over-segment the fundus image using a graph-based image segmentation algorithm to obtain multiple segmented regions of the fundus image;

[0170] The fourth processing unit is used to process the multiple segmented regions according to a selective search algorithm to obtain multiple candidate regions for the target object to be detected.

[0171] Optionally, the fourth processing unit may include:

[0172] The recommended region obtaining unit is used to process the multiple segmented regions according to the selective search algorithm to obtain multiple recommended regions;

[0173] The filtering unit is used to determine that there are overlapping regions among the different recommended regions, and to filter the multiple recommended regions according to the non-maximum suppression algorithm to obtain multiple candidate regions for the target object to be detected.

[0174] Optionally, the feature extraction network in the above embodiments is a pre-trained convolutional neural network, which includes a visual geometry group (VGG16) network. This network is used for weakly supervised training using sample fundus images with image-level category labels to obtain a target detector training model. This model may include:

[0175] The first acquisition unit is used to acquire the category prediction loss of the sample category information of each candidate region in the sample fundus image, and the regression loss of the sample bounding box information respectively.

[0176] The region prediction loss acquisition unit is used to obtain the region prediction loss corresponding to the candidate region of the sample based on the category prediction loss and the regression loss of each candidate region of the sample.

[0177] The cross-entropy loss acquisition unit is used to obtain the corresponding cross-entropy loss based on the prediction loss of each region and the image-level category label of the corresponding sample fundus image;

[0178] A training unit is used to train the model by minimizing the cross-entropy loss to obtain the target detector.

[0179] In some other embodiments, the target detection result acquisition module 105 described above may include:

[0180] The target bounding box information obtaining unit is used to obtain the target bounding box information of the fundus hemorrhage point in the fundus image based on the category information and bounding box information of each of the multiple candidate regions;

[0181] The output unit is used to output the fundus image and display the target bounding box of the fundus hemorrhage point on the fundus image.

[0182] It should be noted that the various modules and units in the above-mentioned device embodiments can all be stored in the memory as program modules. The processor executes the above-mentioned program modules stored in the memory to realize the corresponding functions. The functions realized by each program module and its combination, as well as the technical effects achieved, can be referred to the description of the corresponding part of the above-mentioned method embodiments. This embodiment will not repeat them here.

[0183] The present invention also provides a computer-readable storage medium on which a computer program can be stored, which can be called and loaded by a processor to implement the various steps of the weakly supervised target detection method described in the above embodiments. The specific implementation process can be referred to the description of the corresponding part of the above embodiments, and will not be repeated in this embodiment.

[0184] Reference Figure 11 The above is a schematic diagram of the hardware structure of an optional example of an electronic device suitable for the weakly supervised target detection method proposed in this application, as shown below. Figure 11 As shown, the electronic device may include: a communication interface 111, a memory 112, and a processor 113, wherein:

[0185] The number of communication interface 111, memory 112 and processor 113 can all be at least one, and communication interface 111, memory 112 and processor 113 can all be connected to a communication bus to realize data interaction between them through the communication bus. The specific implementation process can be determined according to the needs of the specific application scenario, and will not be described in detail in this invention.

[0186] The communication interface 111 may include a communication interface that enables data interaction using a wireless communication network, such as a WIFI module, a 5G / 6G (fifth-generation mobile communication network / sixth-generation mobile communication network) module, a GPRS module, etc., to realize wireless communication between the electronic device and other devices, such as acquiring fundus images to be detected, a large number of sample fundus images, and outputting target detection results. Of course, the communication interface 111 may also be a data interface for realizing wired communication, so that the electronic device can connect to other devices through a corresponding type of data cable to realize data transmission between different devices, such as input / output data during the execution of a weakly supervised target detection method.

[0187] It should be understood that the aforementioned communication interface 111 includes communication interfaces that enable data interaction between internal components of an electronic device, such as USB interfaces, serial / parallel ports, multimedia transmission interfaces, etc. This application does not limit the type or number of communication interfaces 111.

[0188] In this embodiment, the memory 112 can be used to store a program that implements the weakly supervised target detection method described in any of the above method embodiments; the processor 113 can load and execute the program stored in the memory 112 to implement each step of the weakly supervised target detection method proposed in any of the above method embodiments. The specific implementation process can be referred to the description of the corresponding part of the corresponding embodiment above, and will not be repeated here.

[0189] In practical applications, memory 112 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. Processor 113 may be a central processing unit (CPU), application-specific integrated circuit (ASIC), digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic device, etc., and the types of memory 112 and processor 113 can be determined according to actual needs.

[0190] It should be understood that, Figure 11 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of the present invention. In practical applications, the electronic device may include more than Figure 11The additional components shown, or combinations of certain components, can be determined according to the product type of the electronic device. When the electronic device is a terminal device, it may also include: an image acquisition component, a display, a speaker, or at least one output component. It may also include, as needed, a microphone, function buttons, or at least one input component to meet different input operations on the electronic device, as well as a sensor module containing multiple sensors, a power supply component, etc., which can be determined according to the application scenario. These will not be listed one by one in this application.

[0191] Finally, it should be noted that the terms "system," "device," "unit," and / or "module" used in this invention are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other words can achieve the same purpose, they can be replaced by other expressions.

[0192] As indicated in this invention and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0193] In the description of the embodiments of the present invention, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, in the description of the embodiments of the present invention, "multiple" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0194] Furthermore, the various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0195] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method based on weak supervision, characterized in that, The method includes: Obtain the fundus image to be examined; The fundus image is input into the target detector to extract the feature map of the fundus image and multiple candidate regions for the target object to be detected; the target detector is obtained by weakly supervised training on sample fundus images with image-level category labels; The feature map and the plurality of candidate regions are processed by the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions; wherein, the feature mapping network includes an attention network, a spatial pyramid pooling network, a feature embedding network and a graph neural network stacked in sequence. The classification and regression networks of the target detector are used to classify and detect each target region feature vector and perform bounding box regression processing to obtain the category information and bounding box information of the corresponding candidate region. Based on the category information and bounding box information of each of the multiple candidate regions, the target detection result in the fundus image is obtained; The step of processing the feature map and the plurality of candidate regions through the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions includes: The feature map and the plurality of candidate regions are input into the feature mapping network of the target detector. The attention network is used to enhance the candidate region features of each candidate region in the feature map to obtain a first region feature vector for each of the plurality of candidate regions. The candidate region features of each candidate region in the feature map are processed by a spatial pyramid pooling network to obtain a second region feature vector of the same preset length for each of the multiple candidate regions. The first region feature vector and the second region feature vector of each of the plurality of candidate regions are input into the feature embedding network, and the dimensionality reduction is performed through the weight matrix of the feature embedding network to obtain the third region feature vector of each of the plurality of candidate regions of the preset length. The feature map and the plurality of candidate regions are input into the graph neural network for unsupervised pooling aggregation to obtain the fourth region feature vector of the preset length for each of the plurality of candidate regions; The feature vectors of the third region and the fourth region are concatenated to obtain the target region feature vectors of each of the multiple candidate regions.

2. The method according to claim 1, characterized in that, The step of inputting the first region feature vector and the second region feature vector of each of the plurality of candidate regions into the feature embedding network, and performing dimensionality reduction processing through the weight matrix of the feature embedding network to obtain the third region feature vector of each of the plurality of candidate regions of the preset length includes: The first region feature vector and the second region feature vector of each of the multiple candidate regions are input into the feature embedding network. The first region feature vector is processed by two fully connected layers of the first size and an activation function layer in sequence to obtain the fifth region feature vector. The second region feature vector is then processed by two fully connected layers of the first size in sequence to obtain the sixth region feature vector. Multiply the feature vector of the fifth region and the feature vector of the sixth region to obtain the feature vector of the seventh region. Superimpose the feature vector of the seventh region and the feature vector of the sixth region to obtain the feature vector of the eighth region. The feature vector of the eighth region is processed sequentially through a fully connected layer of the first size and a fully connected layer of the second size to obtain a third region feature vector of the preset length corresponding to the candidate region.

3. The method according to claim 1, characterized in that, The extraction of feature maps from the fundus image, and multiple candidate regions for the target object to be detected, includes: The feature extraction network of the target detector is used to extract features from the fundus image to obtain the feature map of the fundus image; The target detector's region extraction network is used to extract object regions from the fundus image, obtaining multiple candidate regions for the target object to be detected.

4. The method according to claim 3, characterized in that, The region extraction network of the target detector extracts object regions from the fundus image to obtain multiple candidate regions for the target object to be detected, including: The fundus image is input into the region extraction network of the target detector, and the fundus image is over-segmented using a graph-based image segmentation algorithm to obtain multiple segmented regions of the fundus image; Based on the selective search algorithm, the multiple segmented regions are processed to obtain multiple candidate regions for the target object to be detected.

5. The method according to claim 4, characterized in that, The selective search algorithm is used to process the multiple segmented regions to obtain multiple candidate regions for the target object to be detected, including: Based on the selective search algorithm, the multiple segmented regions are processed to obtain multiple recommended regions; By identifying overlapping regions among the different recommended regions, and using a non-maximum suppression algorithm to filter the multiple recommended regions, multiple candidate regions for the target object to be detected are obtained.

6. The method according to claim 3, characterized in that, The feature extraction network is a pre-trained convolutional neural network, which includes the Visual Geometry Group (VGG16) network. In the process of obtaining the target detector through weakly supervised training using sample fundus images with image-level category labels, the category prediction loss of the sample category information of each candidate region in the sample fundus image and the regression loss of the sample bounding box information are obtained respectively. Based on the category prediction loss and the regression loss of each sample candidate region, the region prediction loss corresponding to the sample candidate region is obtained; Based on the prediction loss for each region and the image-level category label of the corresponding sample fundus image, the corresponding cross-entropy loss is obtained; The target detector is obtained by training the model by minimizing the cross-entropy loss.

7. The method according to any one of claims 1-6, characterized in that, The step of obtaining the target detection result in the fundus image based on the category information and bounding box information of each of the multiple candidate regions includes: Based on the category information and bounding box information of each of the multiple candidate regions, the target bounding box information of the fundus hemorrhage points in the fundus image is obtained; Output the fundus image and display the target bounding box of the fundus hemorrhage point on the fundus image.

8. A target detection device based on weak supervision, characterized in that, The device includes: The fundus image acquisition module is used to acquire fundus images to be detected; An extraction module is used to input the fundus image into a target detector, extract the feature map of the fundus image, and extract multiple candidate regions for the target object to be detected; the target detector is obtained by weakly supervised training of sample fundus images with image-level category labels; The target region feature vector acquisition module is used to process the feature map and the plurality of candidate regions through the feature mapping network of the target detector to obtain the target region feature vectors of each of the plurality of candidate regions; wherein, the feature mapping network includes an attention network, a spatial pyramid pooling network, a feature embedding network and a graph neural network stacked in sequence; The prediction processing module is used to perform classification detection and bounding box regression processing on each feature vector of the target region through the classification network and regression network of the target detector, respectively, to obtain the category information and bounding box information of the corresponding candidate region; The target detection result acquisition module is used to obtain the target detection result in the fundus image based on the category information and bounding box information of each of the multiple candidate regions; Specifically, the target region feature vector acquisition module is used to: input the feature map and the plurality of candidate regions into the feature mapping network of the target detector; enhance the candidate region features of each candidate region in the feature map through an attention network to obtain a first region feature vector for each of the plurality of candidate regions; process the candidate region features of each candidate region in the feature map through a spatial pyramid pooling network to obtain a second region feature vector of the same preset length for each of the plurality of candidate regions; input the first region feature vector and the second region feature vector of each of the plurality of candidate regions into the feature embedding network; perform dimensionality reduction processing through the weight matrix of the feature embedding network to obtain a third region feature vector of the preset length for each of the plurality of candidate regions; input the feature map and the plurality of candidate regions into the graph neural network for unsupervised pooling aggregation processing to obtain a fourth region feature vector of the preset length for each of the plurality of candidate regions; and concatenate the third region feature vector and the fourth region feature vector to obtain the target region feature vector for each of the plurality of candidate regions.

9. An electronic device, characterized in that, The electronic device includes: Communication module; A memory for storing a program that implements the weakly supervised target detection method as described in any one of claims 1-7; A processor for loading and executing a program stored in the memory to implement the weakly supervised target detection method as described in any one of claims 1-7.

Images