ROP lesion detection and partition positioning method based on fundus color photo

Abstract

The invention provides a fundus color photo-based ROP lesion detection and partition positioning method, which independently processes lesion area positioning and anatomical area positioning tasks through a double-branch collaborative detection framework, so as to avoid the problem that the position proportion of macular and optic disc is abnormal due to retinal development difference of premature infants, and improve the detection accuracy of the retina of the premature infants. It is ensured that the circular I region accurately reflects a real anatomical structure; the feature fusion module strengthens lesion area feature response and screens key feature channels to jointly improve the target recognition capability in the complex fundus image; a multi-shape joint loss function forces the network to learn a spatial topological relation between a lesion area and a circular anatomical area so as to eliminate feature redundancy and improve detection sensitivity; therefore, the problem of low accuracy of ROP lesion detection and partition positioning is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing and artificial intelligence, in particular to a ROP lesion detection and partition positioning method based on fundus color photographs. BACKGROUND

[0002] Retinopathy of Prematurity (ROP) is a proliferative retinal vascular disease in premature infants and low birth weight infants. Its progression can be divided into one to five stages. Three or more stages require urgent intervention to prevent retinal detachment and blindness. In clinical diagnosis, ophthalmologists manually assess through binocular indirect ophthalmoscopy or fundus photography, which is highly subjective, inefficient, and lacks positioning accuracy. ROP automatic detection technology focuses on staging classification, but lacks lesion area positioning and anatomical association area definition.

[0003] Specifically, the ROP automatic detection technology detects the macular center coordinate and optic disc boundary position of the fundus image, calculates the distance between the optic disc and the macula based on a predefined ratio, draws a circular region with a fixed radius as the circular I zone centered on the macula, labels the lesion area range with a rectangular bounding box, uses a sliding window scan or a feature classifier to identify abnormal vascular proliferation areas, and outputs the lesion position marked with a rectangular box and the staging result.

[0004] However, there are differences in the development of premature retinas, causing the actual positions of the macula and optic disc to deviate from the standard ratio, resulting in deviations between the indirectly calculated circular I zone and the actual lesion area. When the macula or optic disc is obscured by blood vessels or the image is blurred, the positioning process fails. The rectangular box cannot match the irregular shape of the ROP lesion, resulting in incomplete coverage and easy inclusion of normal tissue. The circular I zone positioning and lesion detection tasks are separate, ignoring spatial correlation, resulting in feature redundancy. Blood vessel obstruction and image noise further reduce the recognition robustness in complex scenarios. In summary, the accuracy of ROP lesion detection and partition positioning is low. SUMMARY

[0005] The present application provides a ROP lesion detection and partition positioning method based on fundus color photographs to solve the problem of low accuracy of ROP lesion detection and partition positioning.

[0006] The present application provides a ROP lesion detection and partition positioning method based on fundus color photographs, comprising:

[0007] Obtain a training sample set, which includes multiple fundus RGB images enhanced by brightness;

[0008] Construct a positioning network based on the training sample set, which includes at least a feature fusion module and a double-branch detection module;

[0009] inputting the image to be positioned into the feature fusion module to output a multi-scale feature map;

[0010] inputting the multi-scale feature map into the double-branch detection module to output a rectangular detection frame through a first branch and a circular detection frame through a second branch;

[0011] obtaining a positioning result based on the rectangular detection frame and the circular detection frame;

[0012] constructing a multi-shape joint loss function based on the rectangular detection frame and the circular detection frame, and updating the positioning network through a loss value calculated by the multi-shape joint loss function.

[0013] In some possible embodiments, the obtaining the training sample set comprises:

[0014] obtaining an eye fundus RGB image;

[0015] inputting an original brightness value of the eye fundus RGB image, and outputting a brightened image through a nonlinear transformation;

[0016] setting a brightness adjustment intensity parameter to generate an enhanced sample image based on the brightened image;

[0017] integrating the enhanced sample image into the training sample set.

[0018] In some possible embodiments, the inputting the image to be positioned into the feature extraction unit to output a feature map comprises:

[0019] obtaining an image to be positioned;

[0020] performing initial feature extraction on the image to be positioned through a convolution layer, and generating a first feature map through normalization processing and a nonlinear activation function;

[0021] performing a down-sampling operation on the first feature map to output a down-sampled feature map;

[0022] inputting the down-sampled feature map into a plurality of serially connected residual stage execution layers to perform feature extraction to output a plurality of residual feature maps, wherein sizes of the plurality of residual feature maps are different;

[0023] determining the feature map based on the residual feature maps.

[0024] In some possible embodiments, the first branch comprises a feature enhancement module, and the feature enhancement module comprises a multi-scale feature fusion unit, a spatial attention unit and a channel attention unit.

[0025] The method further comprises:

[0026] The multi-scale feature fusion unit performs a cross-scale fusion operation on a plurality of residual feature maps and fuses through upsampling and lateral connection to generate a plurality of scale fusion feature maps;

[0027] The spatial attention unit applies spatial attention weights to the multi-scale fusion feature maps to output spatial enhanced feature maps;

[0028] The channel attention unit applies channel attention weights to the multi-scale fusion feature maps to output channel enhanced feature maps.

[0029] In some possible embodiments, the first branch includes a first classification branch and a first regression branch; the method further includes:

[0030] The spatial dimension of the residual feature map is adjusted by a convolution kernel to generate a first size feature map;

[0031] In the first classification branch, feature compression is performed by a continuous channel adjustment convolution layer to output a first target class probability map;

[0032] In the first regression branch, feature conversion is performed by a continuous channel adjustment convolution layer to output a rectangular frame boundary coordinate parameter map;

[0033] Based on the first target class probability and the rectangular frame boundary coordinate parameter, a candidate rectangular detection frame is output.

[0034] In some possible embodiments, the second branch includes a second classification branch and a second regression branch; the method further includes:

[0035] The spatial dimension of the residual feature map is adjusted by a convolution kernel to generate a second size feature map;

[0036] In the second classification branch, feature compression is performed by a continuous channel adjustment convolution layer to output a second target class probability;

[0037] In the second regression branch, feature conversion is performed by a continuous channel adjustment convolution layer to output a circular frame center coordinate and a radius offset;

[0038] Based on the second target class probability, the circular frame center coordinate and the radius offset, a candidate circular detection frame is output.

[0039] In some possible embodiments, the method further includes:

[0040] Based on the rectangular detection frame and the circular detection frame, a rectangular region distance intersection union loss term and a circular region distance intersection union loss term are calculated;

[0041] construct a multi-shape joint loss function based on the rectangular region distance IoU loss term and the circular region distance IoU loss term;

[0042] update the positioning network by a loss value calculated by the multi-shape joint loss function.

[0043] In some possible embodiments, the constructing the multi-shape joint loss function comprises:

[0044] obtain a candidate rectangular detection box;

[0045] calculate a first intersection area and a first union area based on coordinates of the candidate rectangular detection box and coordinates of a rectangular labeled region;

[0046] generate an IoU based on a ratio of the first intersection area to the first union area;

[0047] calculate a Euclidean distance between the candidate rectangular detection box and a center point of the labeled region;

[0048] calculate a diagonal length of a minimum circumscribed rectangle containing the candidate rectangular detection box and the rectangular labeled region;

[0049] fuse the IoU, the center point Euclidean distance, and the diagonal length to generate a rectangular region distance IoU loss term;

[0050] construct a multi-shape joint loss function based on the rectangular region distance IoU loss term and the circular region distance IoU loss term.

[0051] In some possible embodiments, the method further comprises:

[0052] obtain a candidate circular detection box;

[0053] calculate a Euclidean distance between the candidate circular detection box and a center point of a circular labeled region based on a center point coordinate and a radius parameter of the circular labeled region;

[0054] determine a projection component parameter of an intersection common chord based on the radius parameter and the center point Euclidean distance;

[0055] calculate a second intersection area and a second union area according to the projection component parameter and the radius parameter;

[0056] generate an IoU based on a ratio of the second intersection area to the second union area;

[0057] fuse the IoU and a geometric penalty term related to the center point distance to generate a circular region distance IoU loss term.

[0058] In some possible embodiments, the first branch includes a feature enhancement module, and the feature enhancement module includes a multi-scale feature fusion unit, a spatial attention unit, and a channel attention unit.

[0059] The inputting of the multi-scale feature map into the double-branch detection module includes:

[0060] The processing of the candidate rectangular detection frame by the multi-scale feature fusion unit, the spatial attention unit, and the channel attention unit includes outputting a rectangular detection frame.

[0061] And,

[0062] The processing of the candidate circular detection frame by the multi-scale feature fusion unit, the spatial attention unit, and the channel attention unit includes outputting a circular detection frame.

[0063] According to the above technical solution, the application provides a ROP lesion detection and partition positioning method based on fundus color photographs, which includes: obtaining a training sample set, the training sample set including a plurality of fundus RGB images after brightness enhancement; constructing a positioning network through the training sample set, the positioning network including at least a feature fusion module and a double-branch detection module; inputting a to-be-positioned image into the feature fusion module to output a multi-scale feature map; inputting the multi-scale feature map into the double-branch detection module to output a rectangular detection frame through a first branch and a circular detection frame through a second branch; and obtaining a positioning result based on the rectangular detection frame and the circular detection frame.

[0064] The method independently processes lesion area positioning and anatomical area positioning tasks through a double-branch cooperative detection architecture, to avoid the problem of abnormal macula and optic disc position ratio caused by differences in retinal development of premature infants, and to ensure that circular zone I positioning accurately reflects the real anatomical structure; the feature fusion module strengthens lesion area feature response and screens key feature channels, to jointly improve the target recognition capability in complex fundus images; a multi-shape joint loss function forces the network to learn the spatial topological relationship between the lesion area and the circular anatomical area, to eliminate feature redundancy and improve detection sensitivity; and thus the problem of low accuracy of ROP lesion detection and partition positioning is solved. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0066] Figure 1A flowchart of a process for positioning a macular region of a fundus image by an edge detection algorithm is provided for the embodiments of the present application.

[0067] Figure 2 A flowchart of a process for ROP lesion detection and zonal positioning based on a fundus color image is provided for the embodiments of the present application.

[0068] Figure 3 A structural diagram of a positioning network is provided for the embodiments of the present application.

[0069] Figure 4 A diagram of a positioning result is provided for the embodiments of the present application.

[0070] Figure 5 A rectangular diagram for calculating intersection over union of a rectangle is provided for the embodiments of the present application.

[0071] Figure 6 A rectangular diagram for calculating intersection over union of a circle is provided for the embodiments of the present application.

[0072] Figure 7 A diagram of a positioning network update process is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0073] The embodiments will be described in detail below with reference to the drawings. In the following description, the same numbers are used to denote the same elements throughout the drawings. The embodiments described in the following examples do not represent all the implementations consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0074] The lesion progression stages of ROP include multiple stages. In stage 1, a white demarcation line appears at the junction of retinal vascular area and avascular area, which is slightly visible and about 80% of which can self-resolve. In stage 2, the demarcation line is thickened and raised, which is moderately visible and part of which can self-resolve. In stage 3, abnormal neovascularization and fibrous tissue hyperplasia appear at the ridge, which is a threshold lesion. In stage 4, part of the retina is detached (subtypes 4A / 4B). In stage 5, the whole retina is detached.

[0075] The lesion positions include zone I, zone II, and zone III. Zone I (the centermost) is a circular area with the optic disc as the center and the optic disc-macular distance x 2 as the radius (the lesion in this area is the most dangerous). Zone II is an annular area from the outer edge of zone I to the nasal serrata. Zone III is the remaining temporal peripheral retina outside zone II (the risk is the lowest).

[0076] For the division of zone II and zone III, it is difficult for the existing device to capture the position of the serrata, so the existing image can only divide zone I and zone II. In the present embodiment, zone I is divided, and the remaining area is zone II.

[0077] The closer the ROP is to the optic disc location, the more dangerous it is. If the distance is directly calculated, the scale will be different, so it is necessary to use partitioning to mark the scale twice the distance from the center of the macula to the optic disc.

[0078] As Figure 1 shown in the prior art, the macular region of the fundus image is located by an edge detection algorithm, and the center point coordinates are identified by color features (macula appears dark red) combined with morphological filtering. This process relies on manually setting threshold values to segment the image, and requires the premise that the macular region is not blocked by blood vessels and the boundary is clear. The detection result is used as the reference point for subsequent geometric calculations, and its accuracy affects the positioning accuracy of the circular partition.

[0079] Specifically, the optic disc boundary is determined using an ellipse fitting technique. First, the candidate region of the optic disc is segmented by brightness threshold, and then the least squares method is used to fit the ellipse boundary parameters (center point, major axis, and minor axis). This process assumes that the optic disc is a standard ellipse and has a clear contrast with the background, and has limited adaptability to uneven image exposure or optic disc tilt.

[0080] Based on clinical experience, a fixed ratio of the distance between the optic disc and the macula (usually 3 times the diameter of the optic disc) is set. This ratio is used as a core parameter for geometric calculation to derive the radius of circular zone I. This method ignores the differences in retinal development in premature infants and forces individual anatomical variations to match standardized parameters.

[0081] Circular zone I is drawn with the macular center as the center and a fixed radius. The boundary of the circular region does not adjust with the curvature of the fundus, resulting in compression and distortion of the temporal region of the retina and insufficient coverage of the nasal region, which is particularly pronounced in highly myopic fundi.

[0082] The lesion area is uniformly labeled with a rectangular boundary box, which is incompatible with the morphology of ROP lesions. Specifically, exudative lesions are distributed in a radial pattern, and the rectangular box includes a large amount of normal tissue. The new blood vessels grow in a dendritic manner, and the rectangular box cannot fit the bifurcation structure, causing false positive labeling and insufficient coverage of the lesions.

[0083] A sliding window with a fixed step size is used to traverse the fundus image, and a shallow classifier (such as SVM) is used to determine whether there is a lesion in the window. However, the window size is fixed and cannot adapt to the size difference between microaneurysms and new blood vessels. If the step size is too large, small lesions will be missed, and if the step size is too small, the calculation efficiency will be low. The classifier relies on manual feature engineering and has weak generalization ability.

[0084] The existing technology locates circular zone I and detects lesions as independent tasks, first completes anatomical partitioning, and then detects lesions within the partition. There is no feature sharing mechanism between the two, resulting in lesions being detected without considering anatomical constraints, and the positioning result exceeding the actual partition range. Features are repeatedly extracted, wasting computational resources, and increasing noise sensitivity.

[0085] The detection method relying on the underlying features (color / edge) fuses the macular region into the background in the case of image underexposure, blurs the optic disc boundary in the case of motion artifacts, and confuses the lesion features in the case of blood vessel projection. The traditional method lacks a feature enhancement mechanism, and the precision sharply decreases in a noisy environment.

[0086] The ROP lesion detection and partition positioning method based on the fundus color photograph provided in some embodiments of the present application synchronously outputs results through parallel rectangular or circular branches, and any target detection failure does not affect another target. The lesion area is automatically constrained within the anatomical partition, solving the sequential collapse and spatial fragmentation problems of the prior art. Furthermore, the feature fusion module integrates microvessel details and lesion semantics, suppresses the response of the blood vessel occlusion area, and strengthens the pathological feature channel to break through the detection bottleneck in a noisy environment.

[0087] As shown in Figure 2 , the method comprises the following steps:

[0088] S100: Obtain a training sample set.

[0089] The training sample set includes multiple fundus RGB images after brightness enhancement. The training sample set includes multiple retinal image samples. Each sample adjusts the pixel value distribution through a nonlinear transformation to enhance the illumination robustness. The sample set serves as the data basis for model training, and its images cover different pathological stages and imaging conditions to provide diversified input for the positioning network.

[0090] S200: Construct a positioning network through the training sample set.

[0091] The positioning network at least includes a feature fusion module and a double-branch detection module. The feature fusion module is used to extract multi-scale visual features, and the double-branch detection module synchronously processes the lesion positioning and anatomical partition tasks.

[0092] The positioning network learns the retinal feature representation through end-to-end training to output more accurate positioning results.

[0093] S300: Input the image to be positioned into the feature fusion module to output a multi-scale feature map.

[0094] The feature fusion module integrates the local details and global semantics of the retinal image to output a feature representation with spatial hierarchy. The multi-scale feature map is the output representation of the feature fusion module, including visual features at different abstraction levels. The shallow features retain details such as blood vessel texture and exudate edges, and the deep features encode the spatial relationship between the lesion area and anatomical landmarks. This multi-scale characteristic adapts to the detection needs of pathological structures of different sizes.

[0095] S400: input the multi-scale feature map into the double-branch detection module to output a rectangular detection frame through the first branch and a circular detection frame through the second branch.

[0096] In an implementation, the process of outputting the rectangular detection frame by the first branch and the circular detection frame by the second branch is executed synchronously, and at this time, the multi-scale feature map is simultaneously input into the two branches of the double-branch detection module. The first branch outputs the rectangular detection frame, the second branch outputs the circular detection frame, and the two branches share the same input feature but work independently, ensuring the task specificity of lesion positioning and anatomical region positioning.

[0097] In another implementation, the rectangular detection frame or the circular detection frame can also be output separately, that is, the process of outputting the rectangular detection frame by the first branch and the circular detection frame by the second branch is executed asynchronously, for example, the multi-scale feature map is input into the second branch in the process of outputting the rectangular detection frame by the first branch.

[0098] However, it can be understood that the output rectangular detection frame and / or circular detection frame are not affected regardless of which implementation is used.

[0099] The rectangular detection frame is a spatial positioning result output by the first branch, and the lesion region boundary is defined by the upper left corner coordinate and the lower right corner coordinate. This output represents the spatial distribution range of retinal abnormal tissues, covering microaneurysm clusters, neovascularization areas and other pathological structures, and provides geometric basis for lesion staging.

[0100] The circular detection frame is an anatomical partition result output by the second branch, and the key anatomical region (such as circular region I) is defined by the center coordinate and the radius. This output describes the spatial relationship of the optic disc and the macular axis, and provides an anatomical reference framework for lesion positioning.

[0101] The double-branch detection module shares the input feature but outputs independently. The rectangular branch focuses on the boundary positioning of the lesion region, and the circular branch focuses on the spatial partition of the anatomical landmark. The two are optimally associated through a loss function to avoid error accumulation in the traditional serial process.

[0102] S500: obtain a positioning result based on the rectangular detection frame and the circular detection frame.

[0103] Specifically, non-maximum suppression is performed on the rectangular detection frame set, and non-maximum suppression is performed on the circular detection frame set, and the finally retained rectangular frame (lesion region) and circular frame (anatomical partition) are output.

[0104] The non-maximum suppression operation is performed on the candidate region feature vector to eliminate redundant detection boxes with excessive overlap. The non-maximum suppression operation performs spatial redundancy elimination on the candidate region feature vector, and filters the optimal positioning result based on confidence ranking and intersection over union threshold to ensure that the output detection box meets the spatial uniqueness and pathological coverage completeness requirements.

[0105] The effectiveness of the candidate region is verified based on the preset spatial constraint condition, and the final positioning result is output. In this embodiment, the positioning result includes the rectangular boundary of the lesion region and the circular parameter of the anatomical associated region.

[0106] For example, when a new premature infant fundus image is input, the trained positioning network extracts multi-scale features through the feature fusion module. In the double-branch detection module, the first branch predicts three candidate rectangular boxes based on high-resolution features (confidence 0.92, 0.85, 0.78), and the second branch predicts two candidate circular boxes based on semantic features (confidence 0.95, 0.82). After non-maximum suppression, two rectangular boxes (indicating temporal and nasal lesions) and one circular box (indicating the I area range) are retained to form the final positioning result.

[0107] As shown in the positioning result shown in Figure 4 In the figures 4a and 4b, 4a is the image to be positioned, and 4b is the processed image. The circular detection box is the circular I area, which is the posterior pole anatomical partition. The radius is 2x disc-macular distance, and the ROP high-risk area (I area) is labeled. The rectangular detection box is the ROP lesion area label, which selects the abnormal vascular proliferation or avascular area (such as stage III lesion neovascularization). The position is verified whether it invades the high-risk I area (the spatial relationship between the red box and the yellow circle). The optic disc is the central dark area, which is the convergence of retinal nerve fibers with a diameter of about 1.5 mm, and is the spatial positioning reference point.

[0108] The method uses an end-to-end deep learning framework to realize automatic detection and partition positioning of ROP lesions. First, the input fundus image is preprocessed, multi-scale features are extracted through the feature fusion module, and the multi-scale features are input into the double-branch detection module to synchronously output the rectangular detection box of the lesion area and the circular detection box of the anatomically associated area. The multi-shape joint loss function constructed based on the double output drives network optimization, and finally the precise positioning result is obtained through candidate region screening.

[0109] Through the double-branch cooperative detection architecture, the lesion region positioning and anatomical region positioning tasks are independently processed. The circular detection box of the anatomically associated region is directly generated by network learning, rather than being indirectly derived by relying on geometric calculation as in traditional methods, to alleviate the problem of abnormal proportion of macula and optic disc position caused by the difference in premature infant retinal development, and to ensure that the circular I area positioning accurately reflects the true anatomical structure.

[0110] In some embodiments, the obtaining the training sample set comprises:

[0111] obtaining a fundus RGB image;

[0112] inputting original brightness values of the fundus RGB image, and outputting a brightened image through a nonlinear transformation;

[0113] setting a brightness adjustment intensity parameter to generate an enhanced sample image based on the brightened image;

[0114] integrating the enhanced sample image into the training sample set.

[0115] To obtain the training sample set, the stored or real-time obtained original fundus RGB image can be called, the fundus RGB image is a retinal color image collected by a fundus camera device, and includes visual information of three spectral channels of red, green and blue. The fundus RGB image records key anatomical structures such as retinal blood vessel distribution, optic disc morphology and macular area, and the original pixel value represents the tissue reflection characteristics under the illumination condition.

[0116] After reading the fundus RGB image pixel matrix, the mathematical mapping operation of the image pixel brightness value is performed through the nonlinear transformation, and the illumination condition is optimized by reconstructing the pixel value distribution.

[0117] In some embodiments, the nonlinear transformation is implemented through a power function relationship:

[0118]

[0119] wherein x is the input pixel value (0-255), and γ is the intensity parameter.

[0120] The transformation curve has a steep slope in the low brightness area (significantly brightens the dark part), and is close to linear in the high brightness area (retains the details of the bright part), and the RGB three channels are processed synchronously to maintain color balance.

[0121] The brightened image is output after the nonlinear transformation, the pixel value distribution of the brightened image is nonlinearly shifted relative to the original image, the brightened image retains all anatomical structure details of the retina (such as microvascular bifurcation points and hemorrhagic plaque edges), but the visual distinguishability of the dark area (such as the macular fovea) is significantly enhanced, and the overexposure phenomenon of the bright area (such as the optic disc reflection) is inhibited.

[0122] To ensure the illumination diversity of the generated samples, a brightness adjustment intensity parameter is set or adjusted, the brightness adjustment intensity parameter γ is used to control the degree of non-linear transformation, and the value range is [0.3, 0.7]. This parameter determines the shape of the mapping curve. When γ = 0.3, the curve is steep, and a high-contrast brightening effect is produced. When γ = 0.7, the curve is gentle, and it is close to the original brightness distribution. Each enhancement operation randomly selects a γ value from a preset range to ensure the illumination diversity of the generated samples.

[0123] After adjusting the brightness adjustment intensity parameter, the generated enhanced sample image is lossless in terms of anatomical structure, i.e., the blood vessel direction and lesion boundary are consistent with the original image; the illumination distribution is reconstructed, i.e., the dark detail visibility is improved, and the bright detail information is not lost; and the pathological features are strengthened, i.e., the contrast between the exudate highlight area and the background is increased.

[0124] After performing the above processing on a preset number of fundus RGB images, a plurality of enhanced sample images are generated, and the plurality of enhanced sample images are integrated to generate a training sample set.

[0125] The non-linear transformation overcomes the common illumination defects of fundus images by reconstructing the pixel value distribution curve, brightens the dark part, improves the visibility of the capillary blood vessels in the macular area, avoids the missed detection of microaneurysms, suppresses the bright part, preserves the details in the optic disc reflection area, and prevents overexposure of lesion information.

[0126] The random intensity parameter enables the model to adapt to diversified illumination scenes. In the training stage, samples generated with different γ values are used to learn and extract illumination-invariant pathological features, and in actual application, exposure variations caused by differences in processing equipment are stabilized.

[0127] A positioning network is constructed through the training sample set, wherein, as shown in Figure 3 the positioning network includes a feature fusion module and a double-branch detection module. In some embodiments, the image to be positioned is input into the feature fusion module to output a multi-scale feature map, including:

[0128] An image to be positioned is obtained.

[0129] Initial feature extraction is performed on the image to be positioned through a convolution layer, and a first feature map is generated through normalization processing and a non-linear activation function.

[0130] A downsampling operation is performed on the first feature map to output a downsampled feature map.

[0131] The downsampled feature map is input into a plurality of serially connected residual stages to perform hierarchical feature extraction to output a plurality of residual feature maps, and the sizes of the plurality of residual feature maps are different.

[0132] Based on the residual feature maps, the multi-scale feature map is determined.

[0133] The image to be positioned is a retinal RGB image collected by a fundus camera, including complete anatomical information of the optic disc, macula and blood vessel network. The clinical input includes different pathological stage samples, such as typical ROP manifestations such as stage I peripheral avascular zone and stage III neovascular proliferation.

[0134] The image to be positioned is an input image that needs to be positioned for ROP lesions and zones. This image is an unprocessed original fundus RGB color photo, including retinal structure information to be analyzed.

[0135] In some embodiments, the feature fusion module includes a 7x7 convolution layer, a max pooling layer, a residual stage 1, a residual stage 2, a residual stage 3, and a residual stage 4. The convolution layer is a calculation unit in the positioning network that performs initial feature extraction, composed of a set of learnable convolution kernels. By performing sliding window multiplication and accumulation operations on the input image or feature map, it extracts basic visual features (such as edges, textures, and spots) in local regions, converts raw pixel information into more expressive low-level feature representations, and outputs a feature map with a size of 512x512x64.

[0136] The max pooling layer is used for downsampling, and outputs a feature map with a size of 256x256x64. The residual stage 1, residual stage 2, residual stage 3, and residual stage 4 allow the network to learn the residual (i.e., the amount of change) between the input features and the output features. By introducing a jump connection to directly pass the input features to the output end and add them to the features after convolution transformation, the gradient vanishing problem in deep network training is effectively alleviated, allowing the network to be built deeper to extract more abstract and higher-level features.

[0137] Among them, the residual stage 1 outputs a feature map with a size of 256x256x256, the residual stage 2 outputs a feature map with a size of 128x128x512, the residual stage 3 outputs a feature map with a size of 64x64x1024, and the residual stage 4 outputs a feature map with a size of 128x128x256.

[0138] For example, the image to be positioned is input into the convolution layer, and each convolution kernel performs sliding convolution operations on the image. Each convolution kernel is used to detect a specific type of local pattern in the image (such as an edge in a specific direction or a texture in a specific shape). Through this local perception and weight sharing, the convolution layer effectively extracts basic low-level visual features from the original pixels. After the convolution operation is completed, the output result is an intermediate result with a spatial size that may be the same as or slightly smaller than the input image (depending on the step size and padding), but the number of channels is equal to the number of convolution kernels.

[0139] To improve the stability and efficiency of training, normalization is applied to the output of the convolutional layer. Specifically, for all the intermediate feature maps generated within a batch, the mean and standard deviation of all spatial position data in each channel are calculated, respectively. Then, the data in each channel is subtracted by its corresponding mean and divided by its standard deviation to achieve data centralization and scaling. Finally, two learnable parameters (scaling factor γ and offset factor β) are introduced to perform affine transformation on the normalized data, giving the network the ability to restore the original distribution or learn a new distribution to alleviate the internal covariate shift problem during network training.

[0140] After normalization, a nonlinear activation function is applied. In this embodiment, the ReLU function is used, which performs an element-wise operation on the normalized data. If the input value is greater than zero, the value is output; if the input value is less than or equal to zero, zero is output. Nonlinear transformation can introduce nonlinear representation capability to the network, enabling it to fit more complex function relationships. After ReLU activation, negative responses in the feature map are suppressed and positive responses are retained, enhancing the sparsity and discriminability of the features. The feature representation obtained after the convolutional layer output, normalization processing and ReLU activation is the first feature map. The first feature map carries low-level feature information of the image after preliminary conversion.

[0141] To reduce the complexity of subsequent calculations and expand the receptive field while preserving key information, a downsampling operation needs to be performed on the first feature map. In this embodiment, the maximum pooling method is used to achieve downsampling. For example, a fixed-size sliding window (e.g., 2x2 pixels) is set, and the window slides on the first feature map with a fixed step size (e.g., 2). For each local region covered by the window, the maximum value of all pixel points in this region is calculated, and this maximum value is taken as the value of the output downsampled feature map at this position. Maximum pooling has the effect of selecting the most significant feature response and provides invariance to small spatial displacements. After this operation, the spatial dimensions (width and height) of the output downsampled feature map are reduced to half of the first feature map (if a 2x2 window is used and the step size is 2), and the number of channels remains unchanged.

[0142] The downsampled feature map is sent to a series of residual stages for hierarchical feature extraction, and each residual stage is composed of a series of residual blocks stacked in sequence. The input features are first transformed by one or more convolutional layers, and normalization and ReLU activation are applied during the transformation process. At the same time, the input features are directly transmitted through a jump connection. Finally, the convolution-transformed features are added element-wise (Add operation) to the original input features transmitted through the shortcut. If the number of channels or spatial dimensions of the input and output do not match, a 1x1 convolution is used on the jump connection to adjust. This residual learning mechanism enables the positioning network to easily learn the small incremental changes (residuals) from input to output, alleviating the optimization difficulties of deep networks.

[0143] Different residual stages have different spatial down-sampling rates and channel expansion multiples. For example, the first residual stage can keep the spatial size and mainly increase the number of channels; the subsequent residual stages can perform spatial down-sampling once (such as through a convolution with a step of 2) when entering, while significantly increasing the number of channels. The output result after each residual stage processing is the residual feature map of this stage.

[0144] After the step-by-step processing of multiple residual stages, the positioning network outputs a set of residual feature maps with different sizes (width and height) and increasing channel depths. These feature maps constitute a multi-scale and multi-level feature representation of the image from local details to global semantics. These residual feature maps of different sizes are multi-scale feature maps.

[0145] The multi-scale feature maps will be sent to the dual-branch detection module for further processing to provide a rich feature basis for subsequent dual-branch detection tasks. The dual-branch detection module includes a first branch and a second branch, the first branch is used to output a rectangular detection box, and the second branch is used to output a circular detection box.

[0146] In some embodiments, the method further comprises:

[0147] performing spatial dimension adjustment on the residual feature map through a convolution kernel to generate a first size feature map;

[0148] In the first classification branch, performing feature compression through a continuous channel adjustment convolution layer to output a first target class probability map;

[0149] In the first regression branch, performing feature conversion through a continuous channel adjustment convolution layer to output a rectangular frame boundary coordinate parameter map;

[0150] Outputting a candidate rectangular detection box based on the first target class probability and the rectangular frame boundary coordinate parameter.

[0151] In this embodiment, the residual feature map is the feature map output by the third residual stage, and a 3x3 convolution kernel is used to perform processing on the feature map, which is slid on the input feature map, and a certain pattern is extracted or feature transformation is performed through local area product accumulation calculation.

[0152] In some embodiments, the first classification branch includes four 1x1 convolutions and one max-pooling layer, and the first regression branch includes four 1x1 convolutions, that is, the first feature map is processed through four 1x1 convolutions and one max-pooling layer to generate a first size feature map, which is a feature map of 128x128x128.

[0153] Spatial dimension adjustment is an operation that changes the spatial size (width and height) or the number of channels of the input feature map using convolution operation. By setting the stride, padding and the number of convolution kernels, downsampling, size or channel compression or expansion can be achieved.

[0154] The first classification branch is used to predict the probability of the existence of ROP lesions (or other specified categories) at each spatial position (or preset anchor point). The first size feature map input first flows through the continuous channel adjustment convolution layer. In the first layer 1x1 convolution, the input feature map (such as size 128x128x128) is received, and by setting the number of convolution kernels (such as 256), an intermediate feature map (128x128x256) of higher dimension is output, which improves the expression ability of the feature.

[0155] In the second layer 1x1 convolution, the number of convolution kernels is set to K+1 (K is the number of target categories, for example, ROP lesion staging categories; +1 represents background), and then sequentially through the third and fourth layer convolutions, the first target category probability map (size such as 128x128x(K+1)) is output through the fourth layer convolution. Each spatial position (i,j) of this probability map corresponds to a vector of length K+1. The Softmax function is applied to this vector for normalization, and the value of each element in the vector represents the probability of the existence of the corresponding category target (or background) at this position.

[0156] Then, the continuous channel adjustment convolution layer of the first regression branch, i.e., four 1x1 convolutions, is used to predict the accurate position offset of the corresponding rectangular frame relative to the preset anchor frame at each spatial position (or preset anchor point). The first size feature map input (the same as the classification branch) also flows through the continuous channel adjustment convolution layer for feature conversion. In the first layer 1x1 convolution, the structure is the same as the first layer of the classification branch (such as outputting a 128x128x256 feature map), and the shared part of the low-level feature representation is the same. In the second layer 1x1 convolution, the number of convolution kernels is fixed at 4 (corresponding to the four boundary coordinate offsets of the rectangular frame: Δx1, Δy1, Δx2, Δy2), and then through the third and fourth layer convolutions, the rectangular frame boundary coordinate parameter map (size such as 128x128x4) is output. Each spatial position (i,j) of this parameter map corresponds to a vector of length 4, and the vector element value is the predicted coordinate offset.

[0157] Finally, the generation of the candidate rectangular detection frame needs to combine the outputs of the classification branch and the regression branch. For each preset spatial position (or anchor point) on the feature map, the corresponding category probability vector is taken out from the first target category probability map. The category with the highest probability is selected (if the highest probability corresponds to the background, then this anchor point is ignored).

[0158] The four predicted coordinate offsets (△x1, △y1, △x2, △y2) are taken from the rectangular bounding box coordinate parameter map at the same position. Using the preset anchor box (a rectangle with a default center position, width and height) associated with this position and the predicted offsets, the final rectangular box coordinates are calculated using the bounding box regression formula. The predicted box center x coordinate is the sum of the anchor box center x coordinate and △cx times the anchor box width, the predicted box center y coordinate is the sum of the anchor box center y coordinate and △cy times the anchor box height, the predicted box width is the anchor box width times exp(△w), and the predicted box height is the anchor box height times exp(△h).

[0159] The offset definition is directly the corner point offset or the center point offset + the logarithm of the width and height scaling. The embodiment adopts a general bounding box regression.

[0160] Each rectangular box generated by this process, together with its corresponding class label and classification confidence (i.e. prediction probability), constitutes a candidate rectangular detection box. A large number of candidate boxes constitute the preliminary detection result set of the first branch.

[0161] In this embodiment, through spatial dimension adjustment, the input residual feature is further refined and adapted to form a first feature map optimized for rectangular target detection, strengthening the expression of local features related to irregular morphologies such as lesion edges and texture heterogeneity. The first classification branch gradually compresses and converts features through successive channel adjustment convolution layers, finally outputting a class probability map that focuses on learning abstract features to distinguish different ROP lesion stages or to distinguish lesions from backgrounds, ensuring the accuracy of class judgment. At the same time, the first regression branch uses a similar network structure (sharing part of the bottom features) to focus on learning how to accurately adjust the preset anchor box to the boundary position of the real lesion area, allowing the two sub-tasks to be performed independently.

[0162] Finally, by combining the classification prediction results (class label and confidence) at each spatial position with the coordinate offsets predicted by regression, and based on the preset anchor box, the final rectangular coordinates are calculated to generate candidate rectangular detection boxes. This end-to-end prediction method enables the output candidate boxes to have complete information for subsequent screening (such as non-maximum suppression) and final positioning, simplifying the process and improving efficiency.

[0163] In the second branch, a spatial dimension adjustment is performed on the residual feature map through a convolution kernel to generate a second size feature map;

[0164] In the second classification branch, feature compression is performed through successive channel adjustment convolution layers to output a second target class probability;

[0165] In the second regression branch, feature conversion is performed through successive channel adjustment convolution layers to output the circular center coordinates and radius offset of the circular box.

[0166] Output the candidate circular detection frame based on the second target category probability, the circular frame center coordinate and the radius offset.

[0167] The structure of the convolution kernel, the second classification branch and the second regression branch is the same as that of the convolution kernel, the first classification branch and the first regression branch in the first branch, and the processing process is also the same. For details, refer to the first branch, which will not be described here.

[0168] Unlike the first branch, in the second regression branch, the input features are mapped into circular geometric parameters through a continuous channel adjustment convolution layer. Specifically, the second regression branch is used to predict the geometric parameter offset of the circular frame corresponding to each spatial position or a preset anchor point relative to the preset anchor circle at this position. The input second size feature map (the same as the classification branch) flows through the continuous channel adjustment convolution layer for feature conversion.

[0169] In the first layer 1x1 convolution, it is the same as the first layer of the classification branch (such as outputting a 64x64x256 feature map); in the second layer 1x1 convolution, the number of convolution kernels is fixed at 3 (corresponding to the center coordinate offset △cx, △cy and the radius scaling offset △r respectively), and then the third layer and the fourth layer convolution output the center coordinate and the radius offset of the circular frame (size such as 64x64x3). Each spatial position (i, j) of this parameter map corresponds to a vector of length 3 ([△cx, △cy, △r]).

[0170] The generation of the candidate circular detection frame needs to combine the outputs of the classification branch and the regression branch. For each preset spatial position (or anchor point) on the feature map, the corresponding category probability vector is taken out from the second target category probability. The category with the highest probability is selected (if the highest probability corresponds to the background, this anchor point is ignored).

[0171] The predicted three parameters (△cx, △cy, △r) are taken out from the center coordinate and the radius offset map of the circular frame at the same position. Using the preset anchor circle (a circular shape with a default center position and a radius) associated with this position and the predicted offset, the circular frame regression formula is applied to calculate the final circular parameters. The predicted center x coordinate is the sum of the anchor circle center x coordinate and △cxx anchor circle radius, the predicted center y coordinate is the sum of the anchor circle center y coordinate and △cyx anchor circle radius, and the predicted radius is anchor circle radiusxexp(△r).

[0172] Each circular frame generated by this process consists of a candidate circular detection frame, which is composed of its center coordinate (cx, cy), radius r, corresponding category label and classification confidence.

[0173] The candidate rectangular detection frame generated by the first branch and the candidate circular detection frame generated by the second branch are further processed by a feature enhancement module arranged in the first branch and the second branch to generate the rectangular detection frame and the circular detection frame.

[0174] In some embodiments, the first branch and the second branch each include a feature enhancement module, and the feature enhancement module includes a multi-scale feature fusion unit, a spatial attention unit and a channel attention unit.

[0175] The feature enhancement module receives a plurality of residual feature maps of different sizes as input, processes through an internal specific calculation unit, and outputs an enhanced feature map to improve the feature representation ability of the target region and the robustness to interference factors. The multi-scale feature fusion unit is used to integrate cross-scale information of the input plurality of residual feature maps, combine deep high semantic features with shallow high resolution features to generate a fusion feature map including rich details and high-level semantic information, and solve the problem of limited expression ability of single scale features.

[0176] The spatial attention unit is used to learn and generate a spatial dimension weight map, and the weight map has the same size (width and height) as the input feature map. Each value in the weight map indicates the importance of the corresponding spatial position feature, which is used to emphasize the target region feature and suppress irrelevant background or noise interference. The channel attention unit is used to learn and generate a channel dimension weight vector, and the length of the weight vector is the same as the number of channels of the input feature map. Each value indicates the importance of the corresponding feature channel, which is used to dynamically adjust the contribution of different channels.

[0177] The method further includes:

[0178] The multi-scale feature fusion unit performs a cross-scale fusion operation on the plurality of residual feature maps, and fuses through upsampling and horizontal connection to generate a plurality of scale fusion feature maps;

[0179] The spatial attention unit applies spatial attention weights to the multi-scale fusion feature map to output a spatial enhanced feature map;

[0180] The channel attention unit applies channel attention weights to the multi-scale fusion feature map to output a channel enhanced feature map.

[0181] The multi-scale feature fusion unit fuses the residual stage 2, stage 3 and stage 4 output features across layers, generates three scale fusion feature maps through upsampling, i.e. bilinear interpolation ROI and horizontal connection, and the sizes of the three scale fusion feature maps are 128x128x256, 64x64x256 and 32x32x256, respectively. The shallow details and deep semantic information are retained at the same time.

[0182] The spatial attention unit applies a 3x3 convolution and a Sigmoid activation to the scale fusion feature map to generate a spatial attention weight map (same size as the feature map), which enhances the target region features and suppresses the background interference by element multiplication; the channel attention unit performs global average pooling and global maximum pooling on the scale fusion feature map, and generates channel weights through a shared fully connected layer and a Sigmoid activation, which dynamically adjusts the importance of each channel and highlights the key feature channels.

[0183] Through the upsampling and lateral connection operations of the multi-scale feature fusion unit, the high-level semantic understanding (such as the class attribute of the lesion and the overall layout of the partition) contained in the deep features is integrated with the fine spatial details (such as the boundary contour of the lesion and the subtle structure of the blood vessels) preserved in the shallow features, so that the subsequent modules can utilize both global context and local precision information to accurately locate lesions and partitions of different sizes.

[0184] The spatial attention weight map generated by the spatial attention unit can dynamically identify and emphasize the spatial positions related to ROP lesions and key anatomical partitions in the image, effectively suppress the background interference (such as large uniform regions, non-lesion blood vessels, and unevenly illuminated regions) in the fundus image, and concentrate computational resources in important regions. Especially in the face of complex situations such as blurred lesion area edges or partially blocked by blood vessels, the spatial enhanced feature map can provide clearer and more prominent target response, improving the reliability of detection.

[0185] The channel attention unit can automatically evaluate and highlight the feature channels that are most discriminative for the current detection task (distinguishing lesions / non-lesions, locating specific partitions) by analyzing the global statistical response (such as average intensity, maximum response) of each feature channel, while suppressing the contribution of redundant or noisy channels.

[0186] After obtaining the candidate rectangular detection box and the candidate circular detection box, in some embodiments, the rectangular and circular detection results output by the double-branch are subjected to confidence screening (threshold 0.5), and high-confidence targets are retained. Then, the high-confidence targets are input into the feature enhancement module. In some embodiments, the candidate rectangular detection box is processed by the multi-scale feature fusion unit, the spatial attention unit, and the channel attention unit to output a rectangular detection box; and the candidate circular detection box is processed by the multi-scale feature fusion unit, the spatial attention unit, and the channel attention unit to output a circular detection box.

[0187] The input of the feature enhancement module is the candidate detection box (including coordinates and confidence) generated by the first branch or the second branch and its corresponding original image features. For ease of explanation, the candidate detection box includes candidate rectangular detection boxes and candidate circular detection boxes.

[0188] For each image region covered by each candidate bounding box, precise region cropping is performed on the fused feature map output by the multi-scale feature fusion unit, and bilinear interpolation technology is used to solve the coordinate quantization error, ensuring that the extracted features are spatially aligned with the candidate box.

[0189] The cropped region features are input into the spatial attention unit, which calculates the weights of each spatial position in the region (high weights focus on the target, and low weights suppress the background). The weight map is multiplied element-wise with the region features to generate spatially enhanced region features.

[0190] The optimized region features are input into the classification and regression branch to calculate the class confidence and coordinate fine-tuning amount. Then, the spatially enhanced features are input into the channel attention unit, which evaluates the importance of each feature channel and generates a channel weight vector. The weight vector is multiplied with the features in the channel dimension through broadcast multiplication, and the channel-enhanced region features are output. In some embodiments, the detection box is further generated through a Flatten layer, a first FC layer, a second FC layer, and a regression head in sequence.

[0191] Among them, the Flatten layer converts the three-dimensional feature block (50x128x7x7) into a two-dimensional feature matrix, retaining all spatial information within the candidate box while eliminating dimensional barriers. The first FC layer is used for feature compression and primary abstraction, and the second FC layer is used for feature refinement and high-level expression. Deep semantic features are gradually extracted through two layers of nonlinear transformation, and the original visual features are converted into high-discriminative vectors. The 256-dimensional output vector carries the essential feature expression of the candidate box region.

[0192] In the first branch, the regression head is a rectangular branch (50x(4+1+1)), where 4 represents the coordinate offset of the rectangular bounding box, the first 1 represents the target class confidence, and the second 1 represents the positioning quality score.

[0193] In the second branch, the regression head is a circular branch (50x(3+1+1)), where 4 represents the circular parameter offset, the first 1 represents the target class confidence, and the second 1 represents the positioning quality score.

[0194] The regression head decodes the 256-dimensional feature vector into geometric parameters and confidence values that can be directly used for final detection, realizing end-to-end mapping from abstract features to specific detection results.

[0195] It should be noted that in this embodiment, feature enhancement modules are provided in both the first branch and the second branch to simultaneously enhance the candidate rectangular detection box and the candidate circular detection box.

[0196] To improve the positioning accuracy of the positioning network, in some embodiments, based on the rectangular detection box and the circular detection box, a rectangular region distance intersection-over-union loss term and a circular region distance intersection-over-union loss term are calculated;

[0197] Based on the rectangular region distance intersection-over-union loss term and the circular region distance intersection-over-union loss term, a multi-shape joint loss function is constructed;

[0198] The loss value calculated by the multi-shape joint loss function updates the positioning network.

[0199] The multi-shape joint loss function fuses the optimization objectives of rectangular and circular detection boxes, and constructs a comprehensive evaluation index through geometric overlap and spatial position correlation factor. This function drives the network to simultaneously learn the lesion positioning accuracy and anatomical partition accuracy, establishing an internal correlation between the two tasks. The loss function considers spatial overlap and position correlation factor at the same time, providing directional guidance for network parameter optimization.

[0200] The scalar value calculated by the loss function represents the overall deviation degree between the current prediction and the true label. The predicted rectangular or circular detection box is compared with the labeled true region to calculate the rectangular DIoU loss and the circular IoU loss, and the total loss value is obtained by weighted summation.

[0201] The gradient of the loss function with respect to the positioning network parameters is calculated by backpropagation, and the network weights are updated using the gradient descent algorithm. The above process is repeated until the loss converges.

[0202] Based on the generated rectangular detection box and the circular detection box, a multi-shape joint loss function is constructed, which in some embodiments includes:

[0203] Obtain a candidate rectangular detection box;

[0204] Based on the coordinates of the candidate rectangular detection box and the coordinates of the rectangular labeled region, calculate the first intersection area and the first union area;

[0205] Based on the ratio of the first intersection area to the first union area, generate the intersection-over-union ratio;

[0206] Calculate the Euclidean distance between the candidate rectangular detection box and the center point of the labeled region;

[0207] Calculate the diagonal length of the minimum circumscribed rectangle including the candidate rectangular detection box and the rectangular labeled region;

[0208] Fuse the intersection-over-union ratio, the center point Euclidean distance, and the diagonal length to generate a rectangular region distance intersection-over-union loss term;

[0209] Based on the rectangular region distance intersection-over-union loss term and the circular region distance intersection-over-union loss term, a multi-shape joint loss function is constructed.

[0210] As shown in Figure 5 , for the rectangle A and the rectangle B in the figure, the distance intersection-over-union (DIoU) needs to be calculated according to the following steps, and the core is to comprehensively measure the similarity by combining the intersection-over-union (IoU) of the two rectangles, the center point distance and the diagonal length of the minimum circumscribed rectangle.

[0211] First, the rectangular parameter definition is given, and the coordinates of the rectangle A and the rectangle B are represented by the upper left corner (x1, y1) and the lower right corner (x2, y2). The rectangle A is (x A1 , y A1 , x A2 , y A2 ), and the center point is (C Ax , C Ay ), wherein, The rectangle B is (x B1 , y B1 , x B2 , y B2 ), and the center point is (C Bx , C By ), wherein,

[0212] The area of the intersection region is calculated, and the coordinates of the upper left corner and the lower right corner of the intersection region are calculated first. The x-coordinate of the intersection upper left corner is as follows:

[0213] x ∩1 =max(x A1 , x B1 );

[0214] The y-coordinate of the intersection upper left corner is as follows:

[0215] y ∩1 =max(y A1 , y B1 );

[0216] The x-coordinate of the intersection right lower corner is as follows:

[0217] x ∩2 =min(x A2 , x B2 );

[0218] The y-coordinate of the intersection right lower corner is as follows:

[0219] y ∩2 =min(y A2 , y B2 );

[0220] If x ∩1 ≥ x ∩2 or y ∩1 ≥ y ∩2 , then the two rectangles have no intersection, |A∩B| = 0; otherwise, |A∩B| = (x ∩2 -x ∩1 )×(y ∩2 -y ∩1 ).

[0221] The area of the union of two regions, the area of rectangle A is |A| = (x A2 -x A1 )×(y A2 -y A1 ), and the area of rectangle B is |B| = (x B2 -x B1 )×(y B2 -y B1 ), then |A∪B| = |A| + |B| - |A∩B|.

[0222] IoU is calculated by the following formula:

[0223]

[0224] If |A∪B| = 0, then IOU = 0.

[0225] The center point distance d0, i.e. the Euclidean distance between the center points of the two rectangles, is calculated by the following formula:

[0226]

[0227] The minimum circumscribed rectangle diagonal length d can include the minimum circumscribed rectangle of the two rectangles, whose upper left corner coordinates are (x min = min(x A1 , x B1 ), y min = min(y A1 , y B1 )), and the right lower corner coordinates are (x max = min(x A2 , x B2 ), y max = min(y A2 , y B2 )), then the diagonal length

[0228] The DIoU calculation formula of the two rectangles is consistent with the DIoU of the bounding box, i.e. by the following formula:

[0229]

[0230] wherein, the IoU represents the overlapping degree of two rectangles, the relative distance of the center point.

[0231] The combination of the two makes DIoU be able to more comprehensively measure the similarity of two rectangles.

[0232] For the circular region distance intersection over union loss term, in some embodiments, a candidate circular detection box is obtained;

[0233] Based on the center coordinates and the radius parameter of the candidate circular detection box and the circular labeled region, the Euclidean distance between the center of the candidate circular detection box and the center of the circular labeled region is calculated;

[0234] Based on the radius parameter and the Euclidean distance of the center, the projection component parameter of the intersection common chord is determined;

[0235] According to the projection component parameter and the radius parameter, the second intersection area and the second union area are calculated;

[0236] Based on the ratio of the second intersection area and the second union area, the intersection over union is generated;

[0237] The intersection over union related to the geometric penalty term of the center distance is fused to generate the circular region distance intersection over union loss term.

[0238] As shown in the following formula: Figure 6 Firstly, the center position of the label is defined as A (Ax, Ay) and the radius is r A , the predicted center position is B (Bx, By) and the radius is r B , half of the common chord is L y , and the distance from the center to the common chord is L x .

[0239] Next, the area of the intersection region of the two circles is calculated, which consists of two arc areas, each of which is equal to the corresponding sector area minus the corresponding triangle area (composed of the center and the two endpoints of the common chord), that is, see the following formula:

[0240] S = (S 扇形ADE -S ΔADE ) + (S 扇形BDE -S ΔBDE ).

[0241] The sector area derivation The central angle of the sector ADE is α = ∠DAE. In the right triangle ACD, That is

[0242] According to the sector area formula (where θ is the central angle in radian), the area of the sector ADE is:

[0243]

[0244] For circle B, by the same token, the central angle of sector BDE is β = ∠DBE β = ∠DBE, Then the area of sector BDE is:

[0245]

[0246] For ΔADE, this triangle is composed of two congruent right triangles ACD and ACE, whose area is the sum of the areas of the two right triangles, i.e.,

[0247]

[0248] For ΔBDE, by the same token, this triangle is composed of two congruent right triangles BCD and BCE, BC = d - L x whose area is:

[0249]

[0250] Substitute the above sector area and triangle area into the area of the intersection region to form the formula, which can be obtained as:

[0251]

[0252] Let L x represent d, and L x is derived from the relationship between L and d:

[0253]

[0254] Solve this equation, considering that d > 0 and d = L x + CB, take the positive root to get:

[0255]

[0256] Substitute into the preliminary derivation of the area of the intersection region formula to get:

[0257]

[0258] Define the distance between the two centers plus the length of the two radii as d, then

[0259]

[0260] Then calculate the final IOU through the following formula:

[0261] IoU = IoU 矩形+IoU 圆形 .

[0262] As Figure 7 indicated, based on the final IOU, a joint loss function is constructed, the gradient of the loss function to the positioning network parameters is calculated by back propagation, the network weight is updated using the gradient descent algorithm, and the above process is repeated until the loss converges.

[0263] In the method, the network framework of the positioning network takes ResNet101 as a backbone feature extraction network, introduces a feature enhancement module and a double detection branch, and realizes high-precision detection of rectangular, circular and circular-like targets through multi-scale feature fusion, shape perception regression and attention mechanism.

[0264] The similar parts among the embodiments provided in the application can be referred to each other, the specific embodiments provided above are only several examples under the general concept of the application, and do not constitute the limitation of the protection scope of the application. For those skilled in the art, any other embodiments extended according to the application scheme without creative labor are within the protection scope of the application.

Claims

1. A method for detecting and zonal localization of ROP lesions based on fundus color imaging, characterized in that, include: Obtain a training sample set, which includes multiple RGB images of the fundus enhanced with brightness. A localization network is constructed using the training sample set, and the localization network includes at least a feature fusion module and a dual-branch detection module. The image to be located is input into the feature fusion module to output a multi-scale feature map; The multi-scale feature map is input into the dual-branch detection module to output a rectangular detection box through the first branch and a circular detection box through the second branch. Based on the rectangular and circular detection frames, the positioning results are obtained.

2. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 1, characterized in that, The acquisition of the training sample set includes: Obtain RGB images of the fundus; Using the original brightness value of the fundus RGB image as input, a brightened image is output through nonlinear transformation; Set the brightness adjustment intensity parameter to generate an enhanced sample image based on the brightened image; The enhanced sample images are integrated into a training sample set.

3. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 1, characterized in that, The step of inputting the image to be located into the feature fusion module to output a multi-scale feature map includes: Obtain the image to be located; Initial feature extraction is performed on the image to be located through convolutional layers, and a first feature map is generated through normalization and nonlinear activation functions; Perform a downsampling operation on the first feature map to output a downsampled feature map; The downsampled feature map is input into multiple cascaded residual stages to perform hierarchical feature extraction, so as to output multiple residual feature maps with different sizes. The multi-scale feature map is determined based on the residual feature map.

4. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 3, characterized in that, The first branch includes a feature enhancement module, which includes a multi-scale feature fusion unit, a spatial attention unit, and a channel attention unit. The method further includes: The multi-scale feature fusion unit performs cross-scale fusion operations on multiple residual feature maps and generates multi-scale fused feature maps by upsampling and lateral connection. The spatial attention unit applies spatial attention weights to the multi-scale fused feature map, and outputs a spatially enhanced feature map. The channel attention unit applies channel attention weights to the multi-scale fused feature map, and outputs a channel-enhanced feature map.

5. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 1, characterized in that, The first branch includes a first classification branch and a first regression branch; The method further includes: By using convolution kernels, the spatial dimensions of the residual feature map are adjusted to generate a feature map of the first size; In the first classification branch, feature compression is performed by adjusting the convolutional layer through continuous channels to output a probability map of the first target category; In the first regression branch, feature transformation is performed by adjusting the convolutional layer through continuous channels to output a rectangular box boundary coordinate parameter map; Based on the probability of the first target category and the coordinate parameters of the bounding box, a candidate rectangular detection box is output.

6. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 1, characterized in that, The second branch includes a second classification branch and a second regression branch; The method further includes: By using convolution kernels, spatial dimension adjustment is performed on the residual feature map to generate a second-size feature map; In the second classification branch, feature compression is performed by adjusting the convolutional layer through continuous channels to output the probability of the second target class; In the second regression branch, feature transformation is performed by adjusting the convolutional layer through continuous channels to output the center coordinates and radius offset of the circular box; Based on the probability of the second target category, the coordinates of the center of the circular box, and the radius offset, candidate circular detection boxes are output.

7. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 1, characterized in that, The method further includes: Based on the rectangular and circular detection boxes, calculate the distance intersection-union ratio (DIU) loss term for the rectangular region and the distance intersection-union ratio (DIU) loss term for the circular region. A multi-shape joint loss function is constructed based on the rectangular region distance intersection-union ratio loss term and the circular region distance intersection-union ratio loss term. The localization network is updated using the loss value calculated by the multi-shape joint loss function.

8. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 7, characterized in that, The construction of the multi-shape joint loss function includes: Obtain candidate rectangular detection boxes; Based on the coordinates of the candidate rectangular detection box and the coordinates of the rectangular labeled area, calculate the area of ​​the first intersecting region and the area of ​​the first merging region; The intersection-union ratio is generated based on the ratio of the area of ​​the first intersecting region to the area of ​​the first merging region. Calculate the Euclidean distance between the candidate rectangular detection box and the center point of the labeled area; Calculate the diagonal length of the smallest bounding rectangle containing the candidate rectangular detection box and the rectangular annotation area; The intersection-union ratio, the Euclidean distance of the center point, and the diagonal length are combined to generate a rectangular region distance intersection-union ratio loss term; Based on the distance intersection-union ratio (DIU) loss term for the rectangular region and the distance intersection-union ratio (DIU) loss term for the circular region, a multi-shape joint loss function is constructed.

9. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 7, characterized in that, The method further includes: Obtain candidate circular detection boxes; Based on the center coordinates and radius parameters of the candidate circular detection box and the circular annotation area, calculate the Euclidean distance between the center of the candidate circular detection box and the center of the circular annotation area. Based on the radius parameter and the Euclidean distance between the center of the circle, the projection component parameters of the intersecting common chord are determined; Calculate the area of ​​the second intersection region and the area of ​​the second merger region based on the projection component parameters and the radius parameters; The intersection-union ratio is generated based on the ratio of the area of ​​the second intersecting region to the area of ​​the second merging region. By fusing the geometric penalty term related to the intersection-union ratio and the distance to the center of the circle, a circular region distance intersection-union ratio loss term is generated.

10. The method for ROP lesion detection and zonal localization based on fundus color photography according to claim 1, characterized in that, The first branch includes a feature enhancement module, which includes a multi-scale feature fusion unit, a spatial attention unit, and a channel attention unit. The step of inputting the multi-scale feature map into the dual-branch detection module to output a rectangular detection box through the first branch and a circular detection box through the second branch includes: The candidate rectangular detection boxes are processed by the multi-scale feature fusion unit, spatial attention unit, and channel attention unit to output rectangular detection boxes. as well as, The candidate circular detection boxes are processed by the multi-scale feature fusion unit, spatial attention unit, and channel attention unit to output circular detection boxes.

Images