Aβ patch identification method applicable to histological section images
By constructing a second-order object detection network based on ResNet-50 and Enhanced-FPN, and combining RPN and ROI Align, the problem of low recognition accuracy and efficiency in mouse Aβ patch images was solved, achieving efficient and accurate patch recognition results.
Patent Information
- Application Number
- CN202310848453.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-11
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-07-11
AI Technical Summary
Existing technologies struggle to achieve accurate and efficient automated identification of mouse Aβ plaques, especially in image data with high noise levels and varied plaque morphologies. Conventional methods are susceptible to noise and have inconsistent computational efficiency.
A second-order target detection network using ResNet-50 residual network and Enhanced-FPN enhanced feature pyramid structure is adopted. Combined with RPN region candidate network and ROI Align region of interest alignment component, patch prediction boxes are generated through feature extraction and candidate box correction, and three-dimensional coordinate correction is performed to achieve accurate patch identification.
It achieves automated recognition with high accuracy (over 90%) in mouse Aβ patch images, has the ability to resist background noise, adapts to the recognition needs of complex types of image data, and improves recognition efficiency and accuracy.
Smart Images

Figure CN116843663B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for Aβ patch identification applicable to histological slice images. Background Technology
[0002] Aβ (amyloid-β) plaques are a typical pathological feature of Alzheimer's disease (AD). Differences in their distribution and density within the brain's anatomical location influence the pathological condition and manifestations of AD, making them a crucial element in AD-related research. AD mice can rapidly exhibit AD-related amyloid protein pathological changes. Using high-resolution microscopic optical imaging data at the mesoscopic level, whole-brain imaging of Aβ plaques in mice can be performed to observe the distribution changes and development trends of Aβ plaques in the AD mouse brain, while simultaneously obtaining metric information such as plaque number and volume. This provides assistance for research on AD disease mechanisms and early intervention. Therefore, accurate plaque identification from images is key to the reliability and accuracy of related research. Aβ plaques are numerous, widely distributed throughout the brain, and exhibit a wide range of size variations (same as Aβ plaque itself), making identification susceptible to image noise. They also possess multiple characteristics such as variable morphology and diffuse signal distribution, making accurate identification from image data extremely difficult. However, current research specifically targeting Aβ plaque image processing in mice is scarce. Most methods rely on threshold extraction within image processing software for plaque identification, which has limitations in both accuracy and efficiency. In summary, current methods for Aβ plaque identification in mouse mesoscopic images suitable for histological sections can be categorized into three main types. First, manual counting is used. While this method is highly reliable, it is time-consuming and labor-intensive, making it suitable only for small-scale, small-area patch image recognition tasks. Second, image processing software is used, such as Farsight and NeuroGPS, which are used for biomedical image signal recognition, and are adapted for patch recognition. However, this method relies on operator experience and still relies on signal extraction methods such as image thresholding. For patch image data, which has a wide range of signal values and is subject to a lot of background noise, it is difficult to achieve good results. Third, conventional image processing methods are used, such as foreground signal extraction based on thresholding or signal segmentation using watershed, region growing, and morphological operations. These methods mostly rely on single image features and are not effective for complex signals with varying morphologies, such as patches. They are also prone to introducing noise during operation and have inconsistent computational efficiency. Based on the above, there is an urgent need for an automated, efficient, and accurate method for precise Aβ patch recognition in AD mouse image data. Summary of the Invention
[0003] Based on the aforementioned deficiencies in the prior art, this invention provides an Aβ patch identification method applicable to histological slice images, which is relatively accurate and efficient.
[0004] To achieve the above objectives, the present invention provides a method for Aβ patch identification in histological section images, comprising: Step 1: Preprocess the image to be tested to form a two-dimensional input image. The attributes of the two-dimensional input image are matched with the second-order object detection network to be used subsequently. Step 2: The input 2D image is input into the second-order object detection network. The second-order object detection network uses a ResNet-50 residual network as the backbone, an Enhanced-FPN feature pyramid structure, and an RPN region candidate network, an ROI Alignment component, and a network prediction head to form a second-order recognition network. It extracts features from the input 2D image, generates candidate boxes, performs secondary correction on the candidate boxes, and outputs prediction results. The prediction results include patch information contained in each of the input 2D images. The second-order object detection network has been pre-trained. Step 3, post-processing of prediction results, including: If the image to be tested is a two-dimensional image, then proceed to step three-one, restore the prediction results in the single layer to the original image coordinate position according to the data block image number and coordinate information. For the borders of the stitched redundant parts, several borders with an IOU intersection-union ratio > 50% are regarded as the prediction boxes of the same patch and merged. The maximum and minimum values in the x-axis direction and the maximum and minimum values in the y-axis direction of these borders are used as the new border boundary values. At this time, the coordinate information of all patch prediction boxes in the single layer is obtained. If the image to be tested is a 3D image, then based on step 31, step 32 is performed to further correct the coordinate information of the patch prediction box according to the number of layers in the z-axis direction, restoring it to the coordinate position in the 3D coordinate system of the data block range to be tested. After processing all layer prediction results in step 31, z-axis coordinate values x1, y1, x2, y2, z are added to the label information of each patch, where x1 and y1 are the coordinate information of the upper left corner of the coordinate box, x2 and y2 are the coordinate information of the lower right corner of the coordinate box, and z is the layer where the prediction box is located. Using global search, the prediction boxes of each layer of a single patch are integrated to determine the final 3D prediction box. According to the coordinate range, along the layer order in the z-axis direction, candidate boxes in the next layer in the z-axis direction whose border center falls within half the radius outside the border center of the current layer are regarded as candidate boxes of the same patch. At the same time, the border search range is set to three layers in the z-axis direction before and after the current border to expand the search. To avoid the situation where the border is integrated into two patch borders due to missed detection in the intermediate layer, for non-uniform spherical patches, the center is inconsistent with the center of the 3D patch candidate box. For patches whose border radius ratio in the three axes is not in the range of 1±0.3, the patch center point of the 3D prediction box is corrected. The layer with the largest patch cross-sectional area in the z-axis direction is regarded as the layer where the patch center point is located, and the center point of the prediction box of this layer is used as the patch center point. Based on the above operations, the integration of the 2D border to the 3D border is realized, and the final patch prediction box coordinate result x1, y1, x2, y2, z1, z2 is obtained.
[0005] The prediction results, which include the path, name, and corner coordinates (x1, y1, x2, y2) of the image to be tested, are stored in a txt file. Here, x1 and y1 are the coordinates of the upper left corner of the prediction box, x2 and y2 are the coordinates of the lower right corner of the prediction box, and z1 and z2 are the starting and ending layer numbers of the layer distribution along the z-axis of the prediction box. Finally, the prediction results, which include the path, name, and patch prediction box information of the image to be tested, are stored in a txt file.
[0006] In one embodiment, step one involves preprocessing the image to be tested, including: If the image to be tested is a two-dimensional image, the image to be tested is cropped sequentially from the origin along the x-axis and y-axis in two directions according to a preset size sliding window, and splicing redundancy is set in both the x-axis and y-axis directions. For the edge parts of the image that are not larger than the preset size, zero gray value pixels are added along the x-axis and y-axis directions respectively. Finally, each image data block to be tested is numbered, named and stored separately. The image data blocks to be tested obtained by cropping the preset size constitute the input two-dimensional image. The preset size matches the second-order target detection network. If the image to be tested is a three-dimensional image, the image to be tested is first sliced and layered along the z-axis according to the resolution to form multiple layered two-dimensional images. Then, each layered two-dimensional image is cropped from the origin along the x-axis and y-axis according to a sliding window of a preset size, and splicing redundancy is set in both the x-axis and y-axis directions. For the edge parts of the image that are not larger than the preset size, zero grayscale pixels are added along the x-axis and y-axis respectively. Finally, each image data block to be tested is numbered, named and stored separately. The image data blocks to be tested of the preset size obtained by cropping constitute the input two-dimensional image. The preset size matches the second-order object detection network.
[0007] In one embodiment, step one involves preprocessing the image to be tested, including: If the image to be tested is a two-dimensional image, then the image to be tested is downsampled using a bicubic interpolation method so that the resolution of the input two-dimensional image matches the resolution of the second-order object detection network to be used subsequently. If the image to be tested is a three-dimensional image, the image to be tested is first sliced and layered along the z-axis according to the resolution to form multiple layered two-dimensional images. Then, the x-axis and y-axis planes of the image to be tested are downsampled using a bicubic interpolation method to make the resolution of the x-axis and y-axis planes of the image to be tested consistent with the resolution of the z-axis direction, isotropic, and to make the resolution of the input two-dimensional image match the resolution of the second-order object detection network to be used subsequently.
[0008] In one embodiment, in step two, the second-order object detection network is pre-trained, wherein the training process includes: Step 4: Create a labeled dataset for training; Step four includes: Step 41: The x-axis and y-axis planes of the training image data are downsampled using a bicubic interpolation method to make the resolution of the x-axis and y-axis planes of the training image data consistent with the resolution of the z-axis direction and isotropic. The training image data is three-dimensional imaging data of microscopic optical section tomography labeled with DANIR-8c fluorescent probe. Step 42: Starting from the top left corner of the training image data, the data is cropped sequentially to a preset size. For the edge parts of the image that are not larger than the preset size, zero grayscale pixels are added along the x-axis, y-axis, and z-axis respectively. Each cropped training image data block has a preset size. Step 43: Select a first preset number of training image data blocks; Step 44: Use the 3D-slicer annotation tool to annotate the selected training image data blocks. All patches in each training image data block are annotated in the form of a cuboid. The annotation information includes the x-axis, y-axis, and z-axis coordinates of the top left and bottom right corners of a single patch, the storage path and name of the training image data block in which the patch is located, and the annotation information of each patch is stored in a single JSON file. Finally, the annotation information of all patches in each training image data block is summarized and written into a txt file for storage. Steps four and five involve splitting all training image data blocks and annotation information along the x, y, and z axes. The training image data is then re-sliced along these three axes at single-layer intervals of resolution. The annotation information is then summarized layer by layer along the re-slicing direction. For a single training image data block, the images of each layer along the three axes are sequentially split and stored as JPG images. The x-axis and y-axis coordinates of the top-left and bottom-right corner points of the rectangular annotation boxes corresponding to all patches in each layer are then summarized and stored as TXT files. This yields the annotation information corresponding to each layer and axis. The images of each layer and axis, along with their corresponding annotation information, constitute two-dimensional annotated image data. All two-dimensional annotated image data constitute the training annotation dataset.
[0009] In one embodiment, in step four, the training image data is three-dimensional imaging data of microscopic optical section tomography of 5XFAD transgenic mice labeled with the DANIR-8c fluorescent probe. In step four, the selected training image data blocks include data from mice aged 2 months, 3 months, 5 months, 6 months, and 10 months; including the hippocampus, pontine gray matter, and main olfactory bulb; and including the distribution of plaques as sparse, common, or dense.
[0010] In one embodiment, in step two, the second-order object detection network is pre-trained, wherein the training process further includes: Step 5: Select a second preset number of two-dimensional labeled image data and randomly divide them into training set, validation set and test set according to a ratio of 6:1:1. For each dataset, use a separate JSON file to summarize the labeling information of all images in the set and use it together with the corresponding images as input to the network.
[0011] In one embodiment, in step two, the second-order object detection network uses a ResNet-50 residual network as its backbone, employs an Enhanced-FPN feature pyramid structure, and utilizes a second-order recognition network composed of an RPN region candidate network, an ROI Align component, and a network prediction head to extract features from the input two-dimensional image, generate candidate boxes, perform secondary correction on the candidate boxes, and output prediction results, including: Step 21: The ResNet-50 residual network consists of one preprocessing module and four convolutional modules. The preprocessing module includes one convolutional layer and one pooling layer, reducing the image size by half to obtain the first layer feature map. The four convolutional modules are constructed as residual networks with stacked convolutional layers and skip connections, containing 3, 4, 6, and 3 Bottleneck sub-units respectively. Each Bottleneck sub-unit contains three convolutional layers, yielding the second, third, fourth, and fifth layer feature maps respectively. The Enhanced-FPN feature pyramid structure performs feature re-extraction and feature fusion. The second, third, and fourth layer feature maps are downsampled to the same size, then concatenated, subjected to two convolutional operations, and processed by the sigmoid activation function. The resulting image is then upscaled and used as weights in conjunction with the downsampled images. The second, third, and fourth layer feature maps are multiplied together. The corresponding elements of the second, third, and fourth layer feature maps after the three weights are multiplied are added together to obtain the feature re-extracted feature map. This feature re-extracted feature map is added element-wise to the original fifth layer feature map to obtain the fused fifth layer new feature map. The fifth layer new feature map is dimensionally aligned to obtain the fifth layer fused feature map. The original fourth, third, and second layer feature maps are reduced in dimension by 1*1 convolution, upsampled to the same size as the previous layer new feature map, and then fused element-wise with the previous layer new feature map and dimensionally aligned to obtain the fourth, third, and second layer fused feature maps, respectively. The fifth layer fused feature map is upsampled once to obtain the sixth layer fused feature map. The second, third, fourth, fifth, and sixth layer fused feature maps are output. Step 22: The candidate region generation network (RPN) performs a 3*3 convolution operation on the input feature map to eliminate the aliasing effect of upsampling. For each feature point in the feature map, based on the side length of five specific strides of 8, 16, 32, 64, and 128, square anchor boxes are generated with aspect ratios of 0.5, 1, and 2, respectively. A 1*1 convolution is performed on the feature map to classify and determine whether each anchor box contains a patch target. At the same time, another 1*1 convolution is performed on the feature map in parallel to perform box regression to correct the position of the anchor boxes. Non-maximum suppression (NMS) is used to retain the top 50% of the anchor boxes in classification scores, and anchor boxes with an intersection-over-union ratio (IOU) > 70% are merged to form candidate boxes for output. Steps two and three: The ROI Align region of interest alignment component fuses the candidate boxes and the feature maps output by the Enhanced-FPN enhanced feature pyramid structure. It determines which level of feature map each candidate box corresponds to based on the size of the candidate box. It uses bilinear interpolation and max pooling to fuse all candidate boxes with the corresponding feature maps and sets them to a fixed size to facilitate subsequent fully connected operations. Step 24: Next, two fully connected layer modules are connected. Each module includes a fully connected layer and a ReLU activation function. Then, the two fully connected layers are used in parallel. One is used to perform coordinate regression on the candidate boxes, and the other is used to classify the objects in the boxes as foreground or background, so as to achieve secondary correction of the candidate boxes. Finally, the prediction result of the patch prediction box is output.
[0012] In one implementation, during training, for the RPN candidate region generation network, a binary cross-entropy function is used as the foreground and background classification loss function for candidate boxes, and a Smooth-L1 function is used as the loss function for candidate box boundary regression. For the network prediction head, a softmax multi-class cross-entropy function is used as the candidate box classification loss function, and a Smooth-L1 function is used as the loss function for candidate box boundary regression. By adjusting various parameters in network training, including the proportion of training set data, network learning rate, learning rate decay, training stride, training epochs, and the NMS non-maximum suppression coefficient, IOU cross-union coefficient, and ROI retention ratio in each module of the network, multiple debugging and training sessions are conducted, and a comprehensive evaluation and comparison of the model performance is performed to select the optimal network parameters. The trained model at this time is then used as the weight-optimal model.
[0013] The Aβ patch recognition method for histological slice images described in this invention constructs a second-order deep learning network for target detection with excellent small target detection performance. By first generating patch target candidate boxes and then performing secondary correction, it accurately identifies the specific coordinate distribution of Aβ patch signals and the predicted bounding rectangle of the signals in the image. It can perform patch recognition on two-dimensional image data containing patches, has good resistance to background noise, can adapt to the patch recognition needs of complex types of image data, achieves high accuracy in patch recognition, improves the recognition accuracy to over 90%, and is also highly efficient. Attached Figure Description
[0014] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. Furthermore, the shapes and proportions of the components in the drawings are merely illustrative to aid in understanding the invention and are not intended to specifically limit the shapes and proportions of the components. Those skilled in the art, guided by the teachings of this invention, can select various possible shapes and proportions to implement the invention according to specific circumstances. In the drawings: Figure 1 A flowchart illustrating a method for identifying Aβ patches in histological slide images, provided in the first embodiment of the present invention; Figure 2 A flowchart illustrating step four of a method for identifying Aβ patches in histological slice images provided in the first embodiment of the present invention; Figure 3 This is a flowchart illustrating step two of a method for identifying Aβ patches in histological slide images, provided in the first embodiment of the present invention. Figure 4 This is a flowchart illustrating step three of a method for identifying Aβ patches in histological slice images, provided in the first embodiment of the present invention. Detailed Implementation
[0015] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0016] Please see Figure 1 As shown, the first embodiment of the present invention provides a method for Aβ patch identification applicable to histological section images, comprising: Step 1: Preprocess the image to be tested to form a two-dimensional input image. The attributes of the two-dimensional input image are matched with the second-order object detection network to be used subsequently. Step two: The input two-dimensional image is input into the second-order object detection network. The second-order object detection network uses a ResNet-50 residual network as its backbone, an Enhanced-FPN feature pyramid structure, and an RPN region candidate network, an ROI Alignment component, and a network prediction head to form a second-order recognition network. It extracts features from the input two-dimensional image, generates candidate boxes, performs secondary correction on the candidate boxes, and outputs prediction results. The prediction results include patch information contained in each of the input two-dimensional images. The second-order object detection network has been pre-trained.
[0017] This invention provides an Aβ patch recognition method for histological section images. It can identify patches in two-dimensional image data containing Aβ patches (hereinafter referred to as patches, meaning the same as Aβ patches). It exhibits good resistance to background noise, can adapt to the patch recognition needs of complex image data, and achieves a high accuracy rate of over 90%. Furthermore, it achieves largely automated patch recognition without human intervention, solving the problem of requiring significant manpower in past patch recognition methods.
[0018] In one embodiment, step one involves preprocessing the image to be tested, including: If the image to be tested is a two-dimensional image, the image to be tested is cropped sequentially from the origin along the x-axis and y-axis in two directions according to a preset size sliding window, and splicing redundancy is set in both the x-axis and y-axis directions. For the edge parts of the image that are not larger than the preset size, zero gray value pixels are added along the x-axis and y-axis directions respectively. Finally, each image data block to be tested is numbered, named and stored separately. The image data blocks to be tested obtained by cropping the preset size constitute the input two-dimensional image. The preset size matches the second-order target detection network. If the image to be tested is a three-dimensional image, the image to be tested is first sliced and layered along the z-axis according to the resolution to form multiple layered two-dimensional images. Then, each layered two-dimensional image is cropped from the origin along the x-axis and y-axis according to a sliding window of a preset size, and splicing redundancy is set in both the x-axis and y-axis directions. For the edge parts of the image that are not larger than the preset size, zero grayscale pixels are added along the x-axis and y-axis respectively. Finally, each image data block to be tested is numbered, named and stored separately. The image data blocks to be tested of the preset size obtained by cropping constitute the input two-dimensional image. The preset size matches the second-order object detection network.
[0019] Because the images to be tested are generally large, cropping them into smaller data blocks helps improve recognition efficiency. In one specific embodiment, the preset size is 512*512 pixels. The first data block consists of pixels 1 to 512 on the x and y axes, the second data block consists of pixels 501 to 1012 on the x-axis and pixels 1 to 512 on the y-axis, and so on. Setting stitching redundancy when cropping data blocks can meet the needs of synchronous prediction of large-size image blocks, effectively improving operating efficiency, and also facilitating the accurate integration of recognition results. The preset size is related to the image size used during the training of the second-order object detection network; that is, if the image size used during the training of the second-order object detection network is 512*512 pixels, then the image to be tested will also be cropped to 512*512 pixels during preprocessing.
[0020] In one embodiment, step one involves preprocessing the image to be tested, including: If the image to be tested is a two-dimensional image, then the image to be tested is downsampled using a bicubic interpolation method so that the resolution of the input two-dimensional image matches the resolution of the second-order object detection network to be used subsequently. If the image to be tested is a three-dimensional image, the image to be tested is first sliced and layered along the z-axis according to the resolution to form multiple layered two-dimensional images. Then, the x-axis and y-axis planes of the image to be tested are downsampled using a bicubic interpolation method to make the resolution of the x-axis and y-axis planes of the image to be tested consistent with the resolution of the z-axis direction, isotropic, and to make the resolution of the input two-dimensional image match the resolution of the second-order object detection network to be used subsequently.
[0021] Because the resolution of the images to be tested is generally high, to alleviate the memory consumption problem in the recognition calculation of high-resolution images, while ensuring the recognition accuracy of small patches, it is necessary to downsample the image data resolution. Since the resolution of 3D image data is generally lower in the z-axis direction than in the x and y-axis directions, the x and y-axis planes of the image to be tested are downsampled to make the resolution of the x and y-axis planes consistent with the resolution in the z-axis direction, ensuring isotropic properties, before cropping. For 2D images with high resolution, downsampling can be used directly to reduce the resolution. The reduced resolution is determined by the resolution of the images used during the training of the second-order object detection network. That is, the resolution of the images used during the training of the second-order object detection network will be the same as the resolution of the image to be tested during preprocessing, ensuring that the resolution of the input 2D image matches the resolution of the subsequent second-order object detection network.
[0022] In one embodiment, in step two, the second-order object detection network is pre-trained, wherein the training process includes: Step 4: Create a labeled dataset for training; Step four includes: Step 41: The x-axis and y-axis planes of the training image data are downsampled using a bicubic interpolation method to make the resolution of the x-axis and y-axis planes of the training image data consistent with the resolution of the z-axis direction and isotropic. The training image data is three-dimensional imaging data of microscopic optical section tomography labeled with DANIR-8c fluorescent probe. Step 42: Starting from the top left corner of the training image data, the data is cropped sequentially to a preset size. For the edge parts of the image that are not larger than the preset size, zero grayscale pixels are added along the x-axis, y-axis, and z-axis respectively. Each cropped training image data block has a preset size. Step 43: Select a first preset number of training image data blocks; Step 44: Use the 3D-slicer annotation tool to annotate the selected training image data blocks. All patches in each training image data block are annotated in the form of a cuboid. The annotation information includes the x-axis, y-axis, and z-axis coordinates of the top left and bottom right corners of a single patch, the storage path and name of the training image data block in which the patch is located, and the annotation information of each patch is stored in a single JSON file. Finally, the annotation information of all patches in each training image data block is summarized and written into a txt file for storage. Steps four and five involve splitting all training image data blocks and annotation information along the x, y, and z axes. The training image data is then re-sliced along these three axes at single-layer intervals of resolution. The annotation information is then summarized layer by layer along the re-slicing direction. For a single training image data block, the images of each layer along the three axes are sequentially split and stored as JPG images. The x-axis and y-axis coordinates of the top-left and bottom-right corner points of the rectangular annotation boxes corresponding to all patches in each layer are then summarized and stored as TXT files. This yields the annotation information corresponding to each layer and axis. The images of each layer and axis, along with their corresponding annotation information, constitute two-dimensional annotated image data. All two-dimensional annotated image data constitute the training annotation dataset.
[0023] Please see Figure 2 As shown, in one specific embodiment, in step four-one, the training image data is three-dimensional imaging data of microscopic optical section tomography of 5XFAD transgenic mice labeled with the DANIR-8c fluorescent probe, with an image resolution of 0.325*0.325*1 μm.3 In step four, the x-axis and y-axis planes of the training image data are downsampled by a factor of 3.08 to 1*1μm using bicubic interpolation. 2 The resolution level is consistent with the z-axis resolution level. The preset size is 512*512*512 pixels, which is then cropped. In step four, the selected training image data blocks include data from mice aged 2 months, 3 months, 5 months, 6 months, and 10 months; including the hippocampal region (HIP), pontine gray matter (PG), and main olfactory bulb (MOB); including the distribution of patches as sparse, common, and dense, with a first preset number of 22 to handle various situations during actual prediction, meeting the patch recognition needs of different brain regions and mouse ages, and exhibiting good transferability and applicability. In step four, the patch annotation information of all patches in each 512*512*512 pixel training image data block is summarized and written into a txt file for storage. In steps four and five, since the patches are three-dimensional structures, labeling them only on a two-dimensional image can easily overlook the low signal intensity parts at the patch edges or mistake small patches for noise. Therefore, a strategy of reslicing the labeled training image data blocks and assigning two-dimensional layer labels in three principal axis directions (x-axis, y-axis, z-axis) is adopted to achieve the labeling of continuous layer patches, reduce the data labeling difficulty of the recognition task, and at the same time increase the types and amounts of training data and the generalization ability of the network to a certain extent, which helps to improve the robustness of the network.
[0024] In one embodiment, in step two, the second-order object detection network is pre-trained, wherein the training process further includes: Step 5: Select a second preset number of 2D labeled image data sets, and randomly divide them into training, validation, and test sets in a 6:1:1 ratio. For each dataset, use a separate JSON file to summarize the annotation information of all images within the set, and use it along with the corresponding images as input to the network. In one specific embodiment, the second preset number is 11264.
[0025] Please see Figure 3As shown, in one embodiment, in step two, the second-order object detection network uses a ResNet-50 residual network as its backbone, employs an Enhanced-FPN feature pyramid structure, and uses a second-order recognition network composed of an RPN region candidate network, an ROI Align region of interest alignment component, and a network prediction head to extract features from the input two-dimensional image, generate candidate boxes, perform secondary correction on the candidate boxes, and output prediction results, including: Step 21: The ResNet-50 residual network consists of one preprocessing module and four convolutional modules. The preprocessing module includes one convolutional layer and one pooling layer, reducing the image size by half to obtain the first feature map C1. The four convolutional modules are constructed as residual networks with stacked convolutional layers and skip connections, containing 3, 4, 6, and 3 Bottleneck sub-units respectively. Each Bottleneck sub-unit contains 3 convolutional layers, yielding the second, third, fourth, and fifth feature maps C2, C3, C4, and C5 respectively. The Enhanced-FPN feature pyramid structure performs feature re-extraction and feature fusion. The second, third, and fourth feature maps C2, C3, and C4 are downsampled to the same size, then concatenated, subjected to two convolutional operations, and processed by the sigmoid activation function. The resulting W is then updimensionalized and used as weights with the downsampled second, third, and fourth feature maps C2, C3, and C4 respectively. Multiply the second-layer feature map C3 and the fourth-layer feature map C4. Add the corresponding elements of the multiplied second-layer feature map C2, third-layer feature map C3, and fourth-layer feature map C4 to obtain the re-extracted feature map E. Add this re-extracted feature map element-wise to the original fifth-layer feature map C5 to obtain the fused fifth-layer new feature map. Align the dimensions of the fused fifth-layer feature map P5. Then, pass the original fourth-layer feature map C4, third-layer feature map C3, and second-layer feature map C2 respectively... 1*1 convolution dimensionality reduction is performed, upsampling to the same size as the new feature map of the previous layer, then element-wise addition and fusion with the new feature map of the previous layer are performed, and dimension alignment is performed to obtain the fourth fused feature map P4, the third fused feature map P3, and the second fused feature map P2, respectively. The fifth fused feature map P5 is upsampled once to obtain the sixth fused feature map P6. The second fused feature map P2, the third fused feature map P3, the fourth fused feature map P4, the fifth fused feature map P5, and the sixth fused feature map P6 are output. Step 22: The candidate region generation network (RPN) performs a 3*3 convolution operation on the input feature map to eliminate the aliasing effect of upsampling. For each feature point in the feature map, based on the side length of five specific strides of 8, 16, 32, 64, and 128, square anchor boxes are generated with aspect ratios of 0.5, 1, and 2, respectively. A 1*1 convolution (softmax) is performed on the feature map to classify and determine whether each anchor box contains a patch target. At the same time, another 1*1 convolution is performed on the feature map in parallel to perform box regression to correct the position of the anchor boxes (bbox_reg). Non-maximum suppression (NMS) is used to retain the top 50% of the anchor boxes in the classification score, and anchor boxes with an Intersection over Union (IOU) ratio >70% are merged to form candidate box proposals output. Steps 2 and 3: The ROI Align region of interest alignment component fuses the candidate bounding box proposals and the feature maps output by the Enhanced-FPN enhanced feature pyramid structure. It determines which level of feature map each candidate bounding box proposal corresponds to based on its size. It uses bilinear interpolation and max pooling to fuse all candidate bounding box proposals with their corresponding feature maps and sets them to a fixed size to facilitate subsequent fully connected operations. Step 24: Next, two fully connected layer modules are connected. Each module includes a fully connected layer (FClayers) and a ReLU activation function. Then, the two fully connected layers are used in parallel. One is used to perform coordinate regression on the candidate bounding boxes (bbox_pred), and the other is used to classify the objects in the box as foreground or background (softmax+cls_prob) to achieve secondary correction of the candidate bounding boxes. Finally, the prediction results of the patch prediction boxes are output.
[0026] The main purpose of network training is to optimize the weights of each part of the network, reducing the loss function of each module to a low level and maintaining stability in subsequent iterations. In one implementation, during training, for the RPN candidate region generation network, a binary cross-entropy function is used as the loss function for candidate box foreground and background classification, and a Smooth-L1 function is used as the loss function for candidate box boundary regression. For the network prediction head, a softmax multi-class cross-entropy function is used as the candidate box classification loss function, and a Smooth-L1 function is used as the loss function for candidate box boundary regression. By adjusting various parameters in network training, including the proportion of training set data, network learning rate (lr), learning rate decay, training steps, training epochs, and the NMS non-maximum suppression coefficient, IOU cross-union coefficient, and ROI retention ratio in each module of the network, multiple debugging and training cycles are performed, and the model performance is comprehensively evaluated and compared to select the optimal network parameters. The trained model at this point is then used as the weight-optimal model.
[0027] Please see Figure 1 As shown, in one embodiment, the Aβ patch identification method applicable to histological section images further includes: Step 3, post-processing of prediction results, including: If the image to be tested is a two-dimensional image, then proceed to step three-one, restore the prediction results in the single layer to the original image coordinate positions according to the data block image number and coordinate information. For the borders of the stitched redundant parts, several borders with an IOU intersection-union ratio > 50% are regarded as the prediction boxes of the same patch and merged. The maximum and minimum values in the x-axis direction and the maximum and minimum values in the y-axis direction of these borders are used as new border boundary values. At this time, the coordinate information of all patch prediction boxes in the single layer is obtained. The prediction results containing the path, name and corner coordinates x1, y1, x2, y2 of the image to be tested are stored in the form of a txt file, where x1 and y1 are the coordinate information of the upper left corner of the prediction box, and x2 and y2 are the coordinate information of the lower right corner of the prediction box. If the image to be tested is a three-dimensional image, then based on step three-one, step three-two is performed to further correct the coordinate information of the patch prediction box according to the number of layers in the z-axis direction, restoring it to the coordinate position in the three-dimensional coordinate system of the data block range to be tested. After processing all layer prediction results in step three-one, z-axis coordinate values x1, y1, x2, y2, z are added to the label information of each patch, where x1 and y1 are the coordinate information of the upper left corner of the coordinate box, x2 and y2 are the coordinate information of the lower right corner of the coordinate box, and z is the layer where the prediction box is located. Using global search, the prediction boxes of each layer of a single patch are integrated to determine the final three-dimensional prediction box. According to the coordinate range, along the layer order in the z-axis direction, candidate boxes in the next layer in the z-axis direction whose border center falls within half the radius outside the border center of the current layer are regarded as candidate boxes of the same patch (e.g., ...). Figure 4 (As shown in the left part of the middle section), at the same time, the search range of the border is set to three layers in both the front and back directions of the current border along the z-axis, expanding the search range to avoid the border being merged into two patch borders due to missed detection of intermediate layers (such as...). Figure 4 (As shown in the middle part), for non-uniform spherical patches, the center is not consistent with the center of the 3D patch candidate box. For patches whose bounding box radius ratios in the three axes are not within the range of 1±0.3, the patch center point of the 3D prediction box is corrected. The layer with the largest patch cross-sectional area in the z-axis direction is regarded as the layer where the patch center point is located, and the center point of the prediction box of this layer is taken as the patch center point (e.g., ...). Figure 4 As shown in the right part of the middle section, based on the above operations, the integration of the two-dimensional bounding box into the three-dimensional bounding box is realized, and the final patch prediction box coordinate results x1, y1, x2, y2, z1, z2 are obtained. Among them, x1 and y1 are the coordinate information of the upper left corner of the prediction box, x2 and y2 are the coordinate information of the lower right corner of the prediction box, and z1 and z2 are the starting and ending layer numbers of the layer distribution of the prediction box in the z-axis direction. Finally, the prediction results containing the path, name and patch prediction box information of the image to be tested are stored in the form of a txt file.
[0028] The Aβ patch recognition method for histological slice images described in this invention constructs a second-order deep learning network for target detection with excellent small target detection performance. By first generating patch target candidate boxes and then performing secondary correction, it accurately identifies the specific coordinate distribution of Aβ patch signals and the predicted bounding rectangle of the signals in the image. It can perform patch recognition on two-dimensional image data containing patches, has good resistance to background noise, can adapt to the patch recognition needs of complex types of image data, achieves high accuracy in patch recognition, improves the recognition accuracy to over 90%, and is also highly efficient.
[0029] This invention achieves largely automated patch recognition without human intervention, solving the problem of requiring significant manpower in past patch recognition methods. Furthermore, it is applicable not only to patch recognition in two-dimensional single-layer images but also to patch recognition tasks in two-dimensional multi-layer data with continuous z-axis distribution, enabling accurate three-dimensional integration of patch recognition results.
[0030] When training the network model, a strategy of labeling patches in 3D image data blocks and then reslicing the image along the three principal axes (x-axis, y-axis, and z-axis) and assigning 2D layer labels was adopted. This strategy enabled the labeling of continuous layer patches, reducing the data labeling difficulty of the recognition task and increasing the types of training data to a certain extent, which helps improve the robustness of the network. Furthermore, training with diverse patch image data from multiple mouse ages and brain regions met the patch recognition needs of different brain regions and mouse ages, giving the invention good transferability and applicability.
[0031] During recognition, an enhanced feature pyramid architecture is used for key feature extraction in the network. The extracted feature maps are reused to improve the network's feature capture capability. Then, a second-order recognition network is built by combining the region candidate network and the ROI Align component. This fully utilizes the multi-scale feature information of the image and ultimately achieves better results in patch recognition, a typical small target detection task, with a higher patch recognition accuracy than existing methods.
[0032] In the post-processing of prediction results, redundant stitching between images is achieved through intersection-union (IU) judgment, meeting the requirement of synchronous prediction of large-size image blocks, effectively improving running efficiency, and realizing accurate two-dimensional integration of recognition results. Based on this, global search and center point correction are combined, utilizing inter-layer patch recognition information along the z-axis, and employing a merging strategy consistent with actual data statistical results to perform three-dimensional integration of patch recognition results. This solves the problem of frame breaks and inconsistent distribution of patch center points and recognition box center points that easily occur in continuous two-dimensional image target detection networks, enabling the application of the method to patch recognition in continuous three-dimensional z-axis layers.
[0033] This invention applies a second-order object detection network to Aβ patch recognition in mouse brain images, improves network performance by optimizing network components, and further combines pre- and post-processing methods for patch images to achieve superior Aβ patch recognition results compared to existing methods. Furthermore, this invention exhibits good transferability and generalization, capable of handling patch recognition needs of mice of different ages and brain regions. Compared to traditional patch distribution center point recognition, this invention can acquire the coordinate distribution and signal bounding rectangle prediction box in a single step. Simultaneously, it significantly reduces human intervention during use and can be applied to patch recognition in three-dimensional continuous layers.
[0034] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this teaching should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents. For purposes of completeness, all articles and references, including patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended as a waiver of that subject matter, nor should it be construed as an indication that the applicant has not considered that subject matter as part of the disclosed application subject matter.
Claims
1. A method for Aβ patch identification applicable to histological section images, characterized in that, include: Step 1: Preprocess the image to be tested to form a two-dimensional input image. The attributes of the two-dimensional input image are matched with the second-order object detection network to be used subsequently. Step 2: The input 2D image is input into the second-order object detection network. The second-order object detection network uses a ResNet-50 residual network as the backbone, an Enhanced-FPN feature pyramid structure, and an RPN region candidate network, an ROI Alignment component, and a network prediction head to form a second-order recognition network. It extracts features from the input 2D image, generates candidate boxes, performs secondary correction on the candidate boxes, and outputs prediction results. The prediction results include patch information contained in each of the input 2D images. The second-order object detection network has been pre-trained. Step 3, post-processing of prediction results, including: If the image to be tested is a two-dimensional image, then proceed to step three-one, restore the prediction results in the single layer to the original image coordinate position according to the data block image number and coordinate information. For the borders of the stitched redundant parts, several borders with an IOU intersection-union ratio > 50% are regarded as the prediction boxes of the same patch and merged. The maximum and minimum values in the x-axis direction and the maximum and minimum values in the y-axis direction of these borders are used as the new border boundary values. At this time, the coordinate information of all patch prediction boxes in the single layer is obtained. If the image to be tested is a three-dimensional image, then based on step three-one, step three-two is performed to further correct the coordinate information of the patch prediction box according to the number of layers in the z-axis direction, and restore it to the coordinate position in the three-dimensional coordinate system of the range of the data block to be tested. After processing all layer prediction results in step three-one, z-axis coordinate values x1, y1, x2, y2, z are added to each patch label information, where x1 and y1 are the coordinate information of the upper left corner of the coordinate box, x2 and y2 are the coordinate information of the lower right corner of the coordinate box, and z is the layer where the prediction box is located. Using global search, the predicted bounding boxes of each layer for a single patch are integrated to determine the final 3D predicted bounding box. Based on coordinate range and layer order along the z-axis, candidate bounding boxes whose center falls within half the radius of the current layer's bounding box center are considered as candidate bounding boxes for the same patch. Simultaneously, the bounding box search range is set to three layers before and after the current bounding box along the z-axis to expand the search range and avoid the situation where bounding boxes are integrated into two patch bounding boxes due to missed detections in intermediate layers. For non-uniform spherical patches, where the center is inconsistent with the center of the 3D patch candidate bounding box, the patch center point of the 3D predicted bounding box is corrected for patches whose bounding box radius ratios in the three axes are not within the range of 1±0.
3. The layer with the largest patch cross-sectional area in the z-axis direction is considered the layer containing the patch center point, and the center point of the predicted bounding box in this layer is used as the patch center point. Based on the above operations, the 2D bounding box is integrated into the 3D bounding box, obtaining the final patch predicted bounding box coordinates: x1, y1, x2, y2, z1, z2.
2. The method for Aβ patch identification in histological section images as described in claim 1, characterized in that, If the image to be tested is a two-dimensional image, the prediction results containing the path, name, and corner coordinates x1, y1, x2, y2 of the patch prediction box are stored in the form of a txt file. If the image to be tested is a three-dimensional image, the final patch prediction box coordinates x1, y1, x2, y2, z1, z2 are obtained. Where x1 and y1 are the coordinates of the upper left corner of the prediction box, x2 and y2 are the coordinates of the lower right corner of the prediction box, and z1 and z2 are the starting and ending layer numbers of the layer distribution in the z-axis direction of the prediction box. Finally, the prediction results containing the path, name and patch prediction box information of the image to be tested are stored in the form of a txt file.
3. The method for Aβ patch identification in histological section images as described in claim 1, characterized in that, In step one, the image to be tested is preprocessed, including: If the image to be tested is a two-dimensional image, the image to be tested is cropped sequentially from the origin along the x-axis and y-axis in two directions according to a preset size sliding window, and splicing redundancy is set in both the x-axis and y-axis directions. For the edge parts of the image that are not larger than the preset size, zero gray value pixels are added along the x-axis and y-axis directions respectively. Finally, each image data block to be tested is numbered, named and stored separately. The image data blocks to be tested obtained by cropping the preset size constitute the input two-dimensional image. The preset size matches the second-order target detection network. If the image to be tested is a three-dimensional image, the image to be tested is first sliced and layered along the z-axis according to the resolution to form multiple layered two-dimensional images. Then, each layered two-dimensional image is cropped from the origin along the x-axis and y-axis according to a sliding window of a preset size, and splicing redundancy is set in both the x-axis and y-axis directions. For the edge parts of the image that are not larger than the preset size, zero grayscale pixels are added along the x-axis and y-axis respectively. Finally, each image data block to be tested is numbered, named and stored separately. The image data blocks to be tested of the preset size obtained by cropping constitute the input two-dimensional image. The preset size matches the second-order object detection network.
4. The method for Aβ patch identification in histological section images as described in claim 1, characterized in that, In step one, the image to be tested is preprocessed, including: If the image to be tested is a two-dimensional image, then the image to be tested is downsampled using a bicubic interpolation method so that the resolution of the input two-dimensional image matches the resolution of the second-order object detection network to be used subsequently. If the image to be tested is a three-dimensional image, the image to be tested is first sliced and layered along the z-axis according to the resolution to form multiple layered two-dimensional images. Then, the x-axis and y-axis planes of the image to be tested are downsampled using a bicubic interpolation method to make the resolution of the x-axis and y-axis planes of the image to be tested consistent with the resolution of the z-axis direction, isotropic, and to make the resolution of the input two-dimensional image match the resolution of the second-order object detection network to be used subsequently.
5. The method for Aβ patch identification in histological section images as described in claim 1, characterized in that, In step two, the second-order target detection network is pre-trained. The training process includes: Step 4: Create a labeled dataset for training; Step four includes: Step 41: The x-axis and y-axis planes of the training image data are downsampled using a bicubic interpolation method to make the resolution of the x-axis and y-axis planes of the training image data consistent with the resolution of the z-axis direction and isotropic. The training image data is three-dimensional imaging data of microscopic optical section tomography labeled with DANIR-8c fluorescent probe. Step 42: Starting from the top left corner of the training image data, the data is cropped sequentially to a preset size. For the edge parts of the image that are not larger than the preset size, zero grayscale pixels are added along the x-axis, y-axis, and z-axis respectively. Each cropped training image data block has a preset size. Step 43: Select a first preset number of training image data blocks; Step 44: Use the 3D-slicer annotation tool to annotate the selected training image data blocks. All patches in each training image data block are annotated in the form of a cuboid. The annotation information includes the x-axis, y-axis, and z-axis coordinates of the top left and bottom right corners of a single patch, the storage path and name of the training image data block in which the patch is located, and the annotation information of each patch is stored in a single JSON file. Finally, the annotation information of all patches in each training image data block is summarized and written into a txt file for storage. Steps four and five involve splitting all training image data blocks and annotation information along the x, y, and z axes. The training image data is then re-sliced along these three axes at single-layer intervals of resolution. The annotation information is then summarized layer by layer along the re-slicing direction. For a single training image data block, the images of each layer along the three axes are sequentially split and stored as JPG images. The x-axis and y-axis coordinates of the top-left and bottom-right corner points of the rectangular annotation boxes corresponding to all patches in each layer are then summarized and stored as TXT files. This yields the annotation information corresponding to each layer and axis. The images of each layer and axis, along with their corresponding annotation information, constitute two-dimensional annotated image data. All two-dimensional annotated image data constitute the training annotation dataset.
6. The method for Aβ patch identification in histological section images as described in claim 5, characterized in that, In step four, the training image data is three-dimensional imaging data of microscopic optical section tomography of 5XFAD transgenic mice labeled with DANIR-8c fluorescent probe. In step four, the selected training image data blocks include data from mice aged 2 months, 3 months, 5 months, 6 months, and 10 months; including the hippocampus, pontine gray matter, and main olfactory bulb; and including the distribution of plaques as sparse, common, or dense.
7. The method for Aβ patch identification in histological section images as described in claim 5, characterized in that, In step two, the second-order object detection network is pre-trained, and the training process further includes: Step 5: Select a second preset number of two-dimensional labeled image data and randomly divide them into training set, validation set and test set according to a ratio of 6:1:
1. For each dataset, use a separate JSON file to summarize the labeling information of all images in the set and use it together with the corresponding images as input to the network.
8. The method for Aβ patch identification in histological section images as described in claim 1, characterized in that, In step two, the second-order object detection network uses a ResNet-50 residual network as its backbone, employs an Enhanced-FPN feature pyramid structure, and utilizes an RPN region candidate network, an ROI Align component, and a network prediction head to form a second-order recognition network. This network extracts features from the input two-dimensional image, generates candidate boxes, performs secondary correction on the candidate boxes, and outputs prediction results, including: Step 21: The ResNet-50 residual network consists of one preprocessing module and four convolutional modules. The preprocessing module includes one convolutional layer and one pooling layer, reducing the image size by half to obtain the first layer feature map. The four convolutional modules are constructed as residual networks with stacked convolutional layers and skip connections, containing 3, 4, 6, and 3 Bottleneck sub-units respectively. Each Bottleneck sub-unit contains three convolutional layers, yielding the second, third, fourth, and fifth layer feature maps respectively. The Enhanced-FPN feature pyramid structure performs feature re-extraction and feature fusion. The second, third, and fourth layer feature maps are downsampled to the same size, then concatenated, subjected to two convolutional operations, and processed by the sigmoid activation function. The resulting image is then upscaled and used as weights in conjunction with the downsampled images. The second, third, and fourth layer feature maps are multiplied together. The corresponding elements of the second, third, and fourth layer feature maps after the three weights are multiplied are added together to obtain the feature re-extracted feature map. This feature re-extracted feature map is added element-wise to the original fifth layer feature map to obtain the fused fifth layer new feature map. The fifth layer new feature map is dimensionally aligned to obtain the fifth layer fused feature map. The original fourth, third, and second layer feature maps are reduced in dimension by 1*1 convolution, upsampled to the same size as the previous layer new feature map, and then fused element-wise with the previous layer new feature map and dimensionally aligned to obtain the fourth, third, and second layer fused feature maps, respectively. The fifth layer fused feature map is upsampled once to obtain the sixth layer fused feature map. The second, third, fourth, fifth, and sixth layer fused feature maps are output. Step 22: The candidate region generation network (RPN) performs a 3*3 convolution operation on the input feature map to eliminate the aliasing effect of upsampling. For each feature point in the feature map, based on the side length of five specific strides of 8, 16, 32, 64, and 128, square anchor boxes are generated with aspect ratios of 0.5, 1, and 2, respectively. A 1*1 convolution is performed on the feature map to classify and determine whether each anchor box contains a patch target. At the same time, another 1*1 convolution is performed on the feature map in parallel to perform box regression to correct the position of the anchor boxes. Non-maximum suppression (NMS) is used to retain the top 50% of the anchor boxes in classification scores, and anchor boxes with an intersection-over-union ratio (IOU) > 70% are merged to form candidate boxes for output. Steps two and three: The ROI Align region of interest alignment component fuses the candidate boxes and the feature maps output by the Enhanced-FPN enhanced feature pyramid structure. It determines which level of feature map each candidate box corresponds to based on the size of the candidate box. It uses bilinear interpolation and max pooling to fuse all candidate boxes with the corresponding feature maps and sets them to a fixed size to facilitate subsequent fully connected operations. Step 24: Next, two fully connected layer modules are connected. Each module includes a fully connected layer and a ReLU activation function. Then, the two fully connected layers are used in parallel. One is used to perform coordinate regression on the candidate boxes, and the other is used to classify the objects in the boxes as foreground or background, so as to achieve secondary correction of the candidate boxes. Finally, the prediction result of the patch prediction box is output.
9. The method for Aβ patch identification in histological section images as described in claim 8, characterized in that, During training, for the RPN candidate region generation network, the binary cross-entropy function is used as the loss function for candidate box foreground and background classification, and the Smooth-L1 function is used as the loss function for candidate box boundary regression. For the network prediction head, the softmax multi-class cross-entropy function is used as the candidate box classification loss function, and the Smooth-L1 function is used as the loss function for candidate box boundary regression. By adjusting various parameters in the network training, including the proportion of training set data, network learning rate, learning rate decay, training stride, training epochs, and the NMS non-maximum suppression coefficient, IOU cross-union coefficient, and ROI retention ratio in each module of the network, multiple debugging and training were carried out, and the model performance was comprehensively evaluated and compared to select the optimal network parameters. The trained model at this time is used as the weight-optimal model.
Citation Information
Patent Citations
Female upper garment key point identification method
CN111027549A
Method and device for detecting insulator string falling area
CN112651954A
