Organ-like image segmentation method and system

Abstract

The invention discloses an organ-like image segmentation method and system, and relates to the field of medical image processing, and the method comprises the steps: generating graph nodes and adjacent graph nodes based on the high-dimensional feature graphs of a current frame and an adjacent frame; a connection relation between graph nodes and adjacent graph nodes is dynamically generated by using a spatial position constraint principle of an original three-dimensional image, and a graph structure with physical structure consistency is constructed; multi-dimensional information of adjacent graph nodes is transmitted to target graph nodes through graph convolution operation, feature deficiency of a current frame is compensated, three-dimensional neighborhood information of adjacent frames is aggregated through a three-dimensional convolution kernel, and cross-dimensional inter-frame feature complementary fusion is achieved. According to the technical scheme of the invention, the method can achieve the more accurate and robust segmentation effect of a two-dimensional segmentation network in a complex organ-like OCT scene.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and relates to an image segmentation method based on optical coherence tomography, and in particular to an organoid image segmentation method and system. Background Art

[0002] Patient-derived cancer organoids are a culture system with a three-dimensional tissue structure that can faithfully reproduce the molecular characteristics, histopathology, and phenotypic characteristics of patient tumors and are widely used in tumor pathology modeling, drug screening, and precision therapy. However, organoids self-organize in a decentralized manner during the culture process, showing complex tissue morphology at multiple scales, and this morphological feature is closely related to tumor heterogeneity.

[0003] Optical coherence tomography, as a non-invasive three-dimensional imaging technology, is widely used in the three-dimensional morphological analysis and dynamic monitoring of patient-derived cancer organoids due to its high resolution, deep penetration and large field of view. However, organoids show multi-scale dispersed self-organization characteristics during the culture process, and their complex morphology is highly correlated with tumor heterogeneity, which poses a severe challenge to the segmentation accuracy of OCT images. Existing methods have the following significant deficiencies in dealing with such tasks: Although the two-dimensional segmentation network based on single-frame images has the advantages of high computational efficiency and flexible design, it completely ignores the contextual information of adjacent frames in the three-dimensional volume data, resulting in discontinuity and boundary breakage in the segmentation results across frames. In particular, in areas where the contrast between organoids and culture medium is low and speckle noise interference is serious, the limitations of single-frame information further aggravate the segmentation error; the inherent speckle noise, streak artifacts and significant diversity of organoid morphology in OCT imaging put forward strict requirements on the segmentation algorithm. Traditional methods rely on artificially designed features or local context modeling, which are difficult to dynamically adapt to different imaging conditions and changes in organoid status, resulting in large fluctuations in segmentation accuracy and high background mis-segmentation rate. Therefore, the present invention aims to break through the above-mentioned technical bottlenecks and achieve efficient, accurate and scalable three-dimensional organoid segmentation by introducing visual graph neural network and multi-dimensional inter-frame information fusion strategy. Summary of the invention

[0004] Technical Purpose In order to solve the above problems, the purpose of the present invention is to provide an organoid image segmentation method and system, which aims to solve the performance bottlenecks in three-dimensional organoid OCT image segmentation caused by the two-dimensional segmentation method ignoring inter-frame continuity, the excessively high computational resource requirements of the three-dimensional segmentation method, and the scarcity of labeled data through visual graph neural network and multi-dimensional inter-frame information fusion technology. Specifically, the system includes: improving the intra-frame integrity and inter-frame consistency of organoid segmentation; optimizing the cross-frame feature fusion capability of the two-dimensional network without the need for continuous frame annotation; reducing video memory occupancy and computational complexity to achieve efficient processing of large-scale three-dimensional OCT data; and enhancing the robustness of the model to speckle noise, morphological heterogeneity, and image quality degradation under drug intervention.

[0005] Technical Solution In order to achieve the above objectives, the present invention provides an organoid image segmentation method and system, which extracts multi-level features of the current frame and adjacent frames, dynamically constructs a graph structure through three-dimensional convolution, aggregates feature information of multi-dimensional adjacent nodes, replaces the traditional k-nearest neighbor algorithm to reduce computational overhead, and restores high-resolution segmentation results by fusing encoder features through upsampling and jump connection. The network is cascaded with a two-dimensional segmentation network, combined with a composite loss function and a data enhancement strategy, and can achieve high-precision segmentation across patients and drug scenarios with only sparse annotation.

[0006] In a first aspect, the present invention provides an organoid image segmentation method, comprising: Generate graph nodes and neighboring graph nodes based on high-dimensional feature graphs of the current frame and neighboring frames, where the current frame is encoded as a target graph node, and context information of its three-dimensional adjacent area is encoded as a neighboring graph node; The spatial position constraint principle of the original three-dimensional image is used to dynamically generate the connection relationship between the graph nodes and the adjacent graph nodes, and a graph structure with physical structure consistency is constructed; The multi-dimensional information of neighboring graph nodes is transferred to the target graph node through graph convolution operation to compensate for the feature loss of the current frame, and the three-dimensional neighborhood information of neighboring frames is aggregated through the three-dimensional convolution kernel to achieve cross-dimensional inter-frame feature complementary fusion.

[0007] Furthermore, the multidimensional inter-frame information fusion network is sequentially connected to the two-dimensional segmentation network as a preprocessing module, and the enhanced feature map output by the fusion network is input into the two-dimensional segmentation network to improve the intra-frame integrity and inter-frame continuity of organoid segmentation and significantly reduce the computing resource requirements.

[0008] Furthermore, multi-label annotation was performed on the three-dimensional OCT images to distinguish between organs, matrix gel and background areas, and the three-dimensional continuity of the annotations was verified frame by frame. The training set was enhanced through random affine transformation, elastic deformation and combined intensity transformation to enhance the robustness of the model to imaging noise and morphological heterogeneity.

[0009] Furthermore, the three-dimensional OCT image is represented as:

[0010] To enhance the current frame The segmentation performance of d slice stacks of its neighboring frames is selected as:

[0011] Furthermore, the multi-label annotation process includes: Import the 3D OCT data into the annotation tool frame by frame and draw multi-label masks frame by frame; Ensure continuity and consistency of annotation during the annotation process, especially maintaining the continuity of organoid and matrix gel boundaries across frames; In areas of the organoid that are large or morphologically complex, annotations are made with a finer granularity; In the Matrigel area, mark the area that covers the immediate support surrounding the organoid and avoid extending into the background area.

[0012] Furthermore, a lightweight two-dimensional convolutional network is used to extract multi-level feature maps of the current frame and adjacent frames, and the maps are stacked into a three-dimensional tensor. The three-dimensional tensor is aggregated with neighborhood information using a zero-filled three-dimensional convolution operation to generate neighborhood graph nodes that are consistent with the physical space distribution.

[0013] Furthermore, the multi-level feature map Extracted by the following formula:

[0014] In the formula, for The k-th level feature map of ; It is a two-dimensional convolution; contain and The convolution kernel size is 1; The convolution kernel size is 3; is the ReLU activation function; It is the maximum pooling downsampling operator; Furthermore, the multi-frame feature map The stacking is processed into a three-dimensional tensor by the following formula:

[0015] Furthermore, based on the three-dimensional spatial position constraints, the neighboring nodes closely related to the target graph nodes are automatically screened, redundant connections are eliminated, the feature similarity between nodes is represented by the adjacency matrix, and the connection weights are dynamically adjusted to avoid the computational complexity brought by the global search.

[0016] Furthermore, in the graph convolution operation, the connection relationship between neighboring nodes only retains the connection that is closely related to the current node in physical space, and the adjacency matrix is ​​determined by the feature similarity of the neighboring nodes.

[0017] Furthermore, three-dimensional neighborhood information of adjacent frames is aggregated through three-dimensional convolution operations, and complementary fusion of multi-dimensional feature information is achieved based on dynamically generated graph node connection relationships, wherein the dynamic connection relationships are automatically generated through three-dimensional spatial position constraints.

[0018] Furthermore, the three-dimensional neighborhood information of adjacent frames is aggregated by the following formula:

[0019] In the formula, ,Right now , is a three-dimensional convolution with zero padding; for The neighboring graph nodes of .

[0020] Furthermore, a multi-layer perceptron is used to fuse the features of the target graph node and its neighboring nodes, and the update formula is:

[0021] In the formula, is the neighboring node features aggregated by 3D convolution; is a multi-layer perceptron.

[0022] This method significantly reduces the computational complexity of graph construction while improving the efficiency of feature fusion. Through spatial position constraints, the model can automatically generate node connection relationships consistent with the physical structure, avoiding the waste of computing resources caused by global search in traditional methods. In addition, it also enhances the model's generalization ability across patients and drug scenarios, enabling it to adapt to organoid segmentation tasks under different imaging conditions.

[0023] Furthermore, through step-by-step upsampling and skip connection operations, the fused feature map and the encoder multi-level features are channel-concatenated, and the high-resolution segmentation results are restored based on the U-Net architecture to enhance the integrity of the organoid boundaries and the continuity of the internal structure.

[0024] Furthermore, the graph nodes after feature update It is incorporated into the decoder to improve the clarity and completeness of organoid features in the feature map. The decoding formula is as follows:

[0025] In the formula, is the upsampling operation; is a two-dimensional convolution operator; Contains GroupNorm and ReLU. The convolution kernel size is (1, 1); The convolution kernel size is (3, 3); is the ReLU activation function.

[0026] Furthermore, the size of the final output image of the current frame fused with the information of the adjacent frames is the same as the input B-scan.

[0027] Furthermore, the cascade network is suitable for the OCT image segmentation task of patient-derived organoids. It can dynamically adapt to the image contrast loss and noise enhancement caused by drug intervention, and balance the segmentation weights of organoids and background through a composite loss function.

[0028] Furthermore, the model training adopts a composite loss function that combines Dice Loss and cross entropy loss, and performs end-to-end training through the Adam optimizer. The segmentation accuracy is quantified based on the Dice coefficient, background mis-segmentation rate, and inter-frame continuity index, and the three-dimensional convolution kernel size and the number of adjacent frames are optimized.

[0029] Furthermore, the initial learning rate of the model training is 0.00005, and a random seed is set to ensure the repeatability of the experiment.

[0030] Furthermore, the dataset is divided into 20% training set and 80% test set to ensure that the model learns core features with limited training samples. The test set is used to evaluate the performance of the model, focusing on detecting its generalization ability on large-scale data.

[0031] Furthermore, the test set does not perform data augmentation, and only basic operations such as normalization are applied.

[0032] Furthermore, the Dice coefficient is used to evaluate the accuracy of the model in segmenting the organoid region, the background mis-segmentation coefficient is used to evaluate the mis-segmentation of the background region by the model, and the inter-frame continuity index is used to ensure the continuity of cross-frame segmentation.

[0033] This architecture significantly reduces the demand for computing resources while ensuring segmentation accuracy, enabling it to efficiently process large-scale three-dimensional OCT data; through the composite loss function, the model can better balance the segmentation weights of small targets and large targets, improving the overall quality of the segmentation results. Its lightweight design makes it possible to apply the model in real-time clinical scenarios, providing technical support for efficient analysis and precise treatment of organoids.

[0034] In a second aspect, the present invention further provides an organoid image segmentation system, the system is based on the method described in the first aspect, and comprises: The encoder is used to extract the multi-level feature maps of the current frame and adjacent frames and convert them into a three-dimensional tensor. Its input is the current frame. and adjacent frames, the output is a high-dimensional feature map for each frame; The multi-dimensional inter-frame information fusion module is used to aggregate the neighborhood information of the three-dimensional tensor to generate neighboring graph nodes, fuse the neighboring node information in the form of graph convolution, transfer the neighboring frame features to the current frame nodes, and automatically generate node connection relationships based on the three-dimensional spatial position constraints, thereby avoiding the high computational complexity brought by the application of the k-nearest neighbor algorithm. The feature map of each frame is represented as a set of graph nodes. ,in Indicates the number of channels; The decoder module is used to output the enhanced feature map of the current frame through upsampling and feature fusion, and input it into the two-dimensional segmentation network to complete organoid segmentation.

[0035] In a third aspect, the present invention also provides a computer device, comprising a management platform and a memory, wherein the management platform is connected to the memory, the memory is used to store a computer program, and the management platform is used to execute the computer program stored in the memory, so that the computer device performs at least one step of the aforementioned organoid image segmentation method.

[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a management platform, at least one step of the aforementioned organoid image segmentation method is implemented.

[0037] The present invention uses a three-dimensional convolution kernel to aggregate the three-dimensional neighborhood information of adjacent frames, dynamically generates node connection relationships based on physical spatial positions, retains boundary information through zero-filled three-dimensional convolution, and avoids the complexity of manually defining connection rules; MIFFNet is cascaded with a two-dimensional segmentation network, and performance and resource consumption are balanced through lightweight design, and the weight distribution of organoids and background is dynamically adjusted by combining Dice Loss and cross entropy loss. The method and system have a wide range of application scenarios, and can achieve a more accurate and robust segmentation effect of the two-dimensional segmentation network in complex organoid OCT scenarios.

[0038] Beneficial Effects By implementing the organoid image segmentation method and system provided by the present invention, the following technical effects are achieved: (1) The 3D convolution kernel is used to aggregate the 3D neighborhood information of adjacent frames, and the node connection relationship is dynamically generated based on the physical spatial position. The boundary information is retained through zero-filled 3D convolution to avoid the complexity of manually defining connection rules. This method significantly reduces the computational complexity of graph construction and improves the efficiency of feature fusion. Through spatial position constraints, the model can automatically generate node connection relationships consistent with the physical structure, avoiding the waste of computing resources caused by global search in traditional methods. In addition, it also enhances the model's generalization ability across patients and drug scenarios, enabling it to adapt to organoid segmentation tasks under different imaging conditions.

[0039] (2) MIFFNet is cascaded with a two-dimensional segmentation network, and performance and resource consumption are balanced through lightweight design. The weight distribution of organoids and background is dynamically adjusted by combining Dice Loss and cross entropy loss. This architecture significantly reduces the demand for computing resources while ensuring segmentation accuracy, enabling it to efficiently process large-scale three-dimensional OCT data; through the composite loss function, the model can better balance the segmentation weights of small and large targets, improving the overall quality of the segmentation results. Its lightweight design makes it possible to apply the model in real-time clinical scenarios, providing technical support for efficient analysis and precise treatment of organoids. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the above-mentioned organoid image segmentation method and system of the present invention more obvious and easy to understand, the drawings required for use in the specific implementation of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A flow chart showing the organoid image segmentation method; Figure 2 A schematic diagram showing the overall architecture of the multi-dimensional inter-frame information fusion module; Figure 3 Schematic diagram of the overall architecture of the multi-dimensional inter-frame information fusion module and the two-dimensional network cascade. DETAILED DESCRIPTION

[0042] In order to facilitate understanding of the embodiments of the present invention, the abbreviations and key terms that may be involved in the embodiments of the present invention are first explained or defined. For undefined abbreviations or key terms, they are commonly understood by those skilled in the art.

[0043] OCT: optical coherence tomography; Regenovo OCT: optical coherence tomography device; ITK-SNAP: medical image segmentation software; torchio - Python library for efficient processing of 3D medical images for deep learning. RandomAffine: A class of the torchio module used to perform random affine transformations on images; RandomElasticDeformation: A class of the torchio module, used for random elastic deformation transformation for medical image enhancement; B-Scan: Optical coherence tomography 2D cross-sectional view, which shows the depth information (longitudinal) and transverse information (horizontal) of the organoid; ReLU: activation function, rectified linear unit function; Dice Loss: loss function used in image segmentation tasks; Cross-Entropy Loss: Cross entropy loss; Dice coefficient: a statistical indicator used to measure the similarity between two sample sets; MIFFNet: Classification network for feature fusion; TransUnet: a deep learning model that combines the Transformer and U-Net structures; GDNet: Deep learning segmentation model for OCT retinal images; nnUNet 3D: An adaptive medical image segmentation framework; FRNet: Deep learning model for OCT retinal vascular images; Infer-time: inference time; Memory: Memory.

[0044] Embodiment 1: A method and system for organoid image segmentation are provided, as described below.

[0045] Organoid image segmentation method, the method is as follows Figure 1 As shown, including: Step 1: downsample the input image to extract high-dimensional feature information; Step 2: Generate graph nodes and neighboring graph nodes through high-dimensional feature graphs of the current frame and neighboring frames, use the spatial position constraints of the original three-dimensional image to generate the connection relationship between the graph nodes and the neighboring nodes, build a graph structure, and transfer the information in the neighboring graph nodes to the target graph nodes based on the convolution update of the graph structure to compensate for the feature information of the current frame; Step 3: restore the high-dimensional feature information and the fused and updated current frame feature map into an output image, and input the output image into a two-dimensional segmentation network to segment the organoids.

[0046] The input image is an OCT image of a patient-derived organoid acquired by the Regenovo OCT device and manually annotated with multiple labels, specifically including: 1. Annotation tools: a) Use ITK-SNAP open source software to perform multi-label annotation on OCT images, supporting fine pixel-level classification.

[0047] 2. Annotation objects: a) Organoids: manually outline the organoids in the OCT image and mark them as foreground (label value is 1); b) Matrigel: Annotate the Matrigel area around the organoid in the OCT image as the less important area in the background (label value 2); c) Background: The area other than organoids and matrix gel is marked as general background (label value is 0).

[0048] 3. Labeling process: a) Import the 3D OCT data into the annotation tool frame by frame and draw multi-label masks frame by frame; b) Ensure continuity and consistency of annotations, especially the coherence of organoid and Matrigel boundaries across frames; c) In areas of large or morphologically complex organoids, a finer annotation granularity is used; d) In the Matrigel area, the annotation should cover the immediate support area surrounding the organoid and avoid extending into the background area.

[0049] 4. Annotation verification: a) Expert review of the annotation results to ensure the accuracy of the boundaries and classification of organoids and matrix gel; b) Use 3D visualization inspection tools to verify the three-dimensional continuity of the annotation.

[0050] 5. Annotation output: a) Output a 3D mask file with multiple labels, where each pixel corresponds to a category label (0: background, 1: organoid, 2: matrix gel); b) Each file contains the original OCT image and the corresponding multi-label annotations.

[0051] The labeled dataset is divided into 20% training set and 80% test set to ensure that the model learns the core features under limited training samples. The test set is used to evaluate the performance of the model, focusing on its generalization ability on large-scale data, including: 1. Data collation standards: a) 3D organoid image data collected from the Regenovo OCT device, including OCT image samples from multiple patients (e.g., Patient 1, Patient 2, etc.); b) The dataset remains complete and no images are excluded to ensure the richness of the data and the generalization ability of the model; c) Each data sample includes: i. Three-dimensional OCT image (resolution 512 × 800 × 800, length × width × number of sections); ii. The corresponding multi-label annotation file (including organoids, matrix gel and background areas).

[0052] 2. Data division basis: a) The data of patient 1 and patient 2 are divided in a ratio of 20%:80%: i. Training set: 20% of the data of patients 1 and 2; ii. Test set: 80% of the data of patient 1 and patient 2.

[0053] b) After division, the training set is mainly used for model training, and the test set is used for model performance evaluation, focusing on detecting its generalization ability on large-scale data.

[0054] Before training the OCT images, the training set is subjected to data preprocessing, wherein the data preprocessing includes at least one of normalization, spatial transformation, random flipping and intensity transformation, and specifically includes: 1. Spatial transformation: Use the torchio data augmentation toolkit to apply spatial transformation operations to OCT images, including: a) Random affine transformation: Apply rotation and scaling transformations to the image with a probability of 0.5 and a parameter range of: i. The rotation angle range is [-10°, 10°]; ii. The scaling factor range is [0.9, 1.1].

[0055] b) Random elastic deformation: Apply elastic deformation to the image through control point interpolation, control parameters: i. Lock the boundary to limit the pixels in the edge area of ​​the image from being affected by elastic deformation; ii. The number of control points is 4, which generates moderate elastic distortion.

[0056] c) Comprehensive spatial transformation: Randomly select one from RandomAffine and RandomElasticDeformation with a probability of 70% and apply it to the image.

[0057] 2. Random flip: a) Randomly flip the image horizontally, vertically or in depth, with a flip probability of 0.5; b) This operation can enhance the robustness of the model to different morphologies of organoids.

[0058] 3. Intensity transformation: a) applying random transformations of intensity levels to simulate variations in imaging and enhance the model’s adaptability to varying image qualities; b) Random gamma transform: nonlinearly adjust the intensity of image pixel values; c) Random blur: Gaussian blur is applied to the image to simulate blurring artifacts in OCT imaging; d) Random deviation field: Add a deviation field to the image to simulate the inhomogeneity that may be introduced by the imaging device; e) Combined intensity transformation: Randomly select one of the above three intensity transformations with a probability of 70% and apply it to the image.

[0059] 4. Application to image data: a) Apply the above preprocessing pipeline to the divided training set data sample by sample; b) No data augmentation is performed on the test set, only basic operations such as normalization are applied.

[0060] The input image is specifically a three-dimensional OCT image containing multiple B-Scans , expressed as:

[0061] To enhance the current frame The segmentation performance of d slice stacks of its neighboring frames is selected as:

[0062] The spatial transformation is to perform random affine transformation, random elastic deformation and comprehensive spatial transformation operations on the OCT image through the torchio data enhancement toolkit.

[0063] The random flipping is to randomly flip the image along the horizontal, vertical or depth direction.

[0064] The intensity transform is a random transform that applies intensity levels to simulate changes in imaging, including random gamma transforms, random blur, random deviation fields, and combined intensity transforms.

[0065] The test set does not perform data augmentation, and only basic operations such as normalization are applied.

[0066] In step 1, features are extracted through a lightweight 2D convolutional network, and multi-scale 3D context information is captured based on slice stacks. The multi-level feature map of each frame Extracted by the following formula:

[0067] In the formula, for The k-th level feature map of ; It is a two-dimensional convolution; contain and The convolution kernel size is 1; The convolution kernel size is 3; is the ReLU activation function; It is the maximum pooling downsampling operator; Multi-frame feature map The stacking is processed into a three-dimensional tensor by the following formula:

[0068] In step 2, the generation of graph nodes and adjacent graph nodes through high-dimensional feature graphs of the current frame and adjacent frames is specifically as follows: given the current frame as input, the current frame is encoded as a graph node, and the information of its three-dimensional adjacent area is encoded as adjacent graph nodes.

[0069] The three-dimensional neighborhood information of adjacent frames is aggregated through a three-dimensional convolution operation, and the complementary fusion of multi-dimensional feature information is achieved based on a dynamically generated graph node connection relationship, wherein the dynamic connection relationship is automatically generated through three-dimensional spatial position constraints.

[0070] The 3D neighborhood information of adjacent frames is aggregated by the following formula:

[0071] In the formula, ,Right now , is a three-dimensional convolution with zero padding; for The neighboring graph nodes of .

[0072] In the graph convolution operation, the connection relationship of neighboring nodes only retains the connection that is closely related to the current node in physical space. The connection relationship of the neighboring nodes can be represented by an adjacency matrix, and the adjacency matrix is ​​determined by the feature similarity of the neighboring nodes.

[0073] For each graph node , merge neighboring nodes The multi-dimensional information of is updated through graph convolution operation:

[0074] In the formula, It is a multi-layer perceptron used to update the feature representation of graph nodes.

[0075] Through upsampling and skip connection operations, channel cascading is performed based on feature maps at different levels and the fused feature maps to enhance the boundary integrity and internal structure continuity of the segmentation results.

[0076] The low-resolution feature map is gradually restored to the same resolution as the input image by upsampling step by step, and the graph nodes after feature update are It is incorporated into the decoder to improve the clarity and completeness of organoid features in the feature map. The decoding formula is as follows:

[0077] In the formula, is the upsampling operation; is a two-dimensional convolution operator; Contains GroupNorm and ReLU. The convolution kernel size is (1, 1); The convolution kernel size is (3, 3); is the ReLU activation function.

[0078] The final output image size of the current frame fused with the information of adjacent frames is the same as the input B-scan.

[0079] The network was trained end-to-end using the Dice Loss and Cross-Entropy Loss composite loss functions, and the weight balance between organoids and matrix gel was achieved based on the Adam optimizer and a batch size of 4. The initial learning rate was 0.00005, and a random seed was set to ensure the repeatability of the experiment.

[0080] The evaluation indicators of the model include Dice similarity coefficient, background mis-segmentation coefficient, inter-frame continuity index, etc.; the Dice coefficient is used to evaluate the segmentation accuracy of the model for the organoid area, the background mis-segmentation coefficient is used to evaluate the mis-segmentation of the model for the background area, and the inter-frame continuity index is used to ensure the continuity of cross-frame segmentation.

[0081] An organoid image segmentation system, the system is based on the aforementioned method, comprising: The encoder is used to extract the multi-level feature maps of the current frame and adjacent frames and convert them into a three-dimensional tensor. Its input is the current frame. and adjacent frames, the output is a high-dimensional feature map for each frame; The overall architecture of the multi-dimensional frame information fusion module is as follows: Figure 2 As shown, the overall architecture of the two-dimensional network cascade is as follows Figure 3 As shown, it is used to aggregate the neighborhood information of the three-dimensional tensor to generate neighboring graph nodes, fuse the neighboring node information in the form of graph convolution, transfer the neighboring frame features to the current frame nodes, and automatically generate node connection relationships based on the three-dimensional spatial position constraints, thereby avoiding the high computational complexity brought by the application of the k-nearest neighbor algorithm. The feature map of each frame is represented as a set of graph nodes ,in Indicates the number of channels; The decoder is used to output the enhanced feature map of the current frame through upsampling and feature fusion, which is input into the two-dimensional segmentation network to complete organoid segmentation.

[0082] Embodiment 2: Based on the above embodiments, the MIFFNet module is sequentially cascaded with two-dimensional segmentation networks such as TransUnet and GDNet.

[0083] The sequential cascade connection method includes: MIFFNet is used as a preprocessing module and sequentially connected to the 2D segmentation network; The MIFFNet receives the OCT images of the current frame and its adjacent frames, and generates an enhanced feature map of the current frame through multi-dimensional inter-frame information fusion; The enhanced current frame feature map is input into a two-dimensional segmentation network such as TransUnet or GDNet for subsequent segmentation tasks.

[0084] This method significantly improves the accuracy and consistency of segmentation results while keeping the network structure simple. The design of the MIFFNet module follows the principle of lightweight, using a single 3D convolution to replace the k-nearest neighbor algorithm graph with high computational complexity; by controlling the number of adjacent frames and the size of the 3D convolution kernel, the system achieves a balance between performance and computational overhead; by aggregating adjacent frame information, the accuracy of organoid boundary and detail segmentation is improved, and multi-dimensional inter-frame fusion is used to reduce the inconsistency between adjacent frames. In addition, it can also enhance the robustness of segmentation when the image contrast decreases and the noise increases due to drug effects.

[0085] Embodiment 3: Based on the aforementioned embodiments, the performance of the visual graph neural network is verified.

[0086] Read the labeled organoid group 3D OCT image dataset from the storage path in advance. The dataset contains organoid OCT images of multiple patients, multiple cancer types, and multiple drugs. Each sample consists of multiple B-scans with a size of .

[0087] Assume that the patients are numbered P1 and P2, the cancer types include pancreatic cancer and bile duct cancer, the experimental conditions are the organoid state after normal culture and drug action, the image contrast is enhanced by normalizing each frame of the B-scan image, and the image is enhanced by data enhancement, while ensuring that the enhanced image retains the organoid boundary features.

[0088] Table 1. Comparison of experimental results of organoid OCT images in multiple patients, multiple cancer types, and multiple drugs

[0089] As shown in Table 1, the pancreatic cancer organoid OCT dataset of patient 1 was cultured normally, and the intrahepatic cholangiocarcinoma OCT dataset of patient 2 contained two drug conditions. In the pancreatic cancer organoid OCT data of patient 1 without drug effect, the 2D segmentation network surpassed the pure 3D network nnUNet 3D trained with sparse annotation dataset in segmentation accuracy, and the background mis-segmentation probability of all 2D networks was less than 10%; when TransUnet was connected in series with MIFFNet, the segmentation accuracy of the xz section was improved by 1%, when FRNet was connected in series with MIFFNet, the intra-frame segmentation accuracy was reduced by 2%, and the background mis-segmentation probability was reduced by 4%, and when GDNet was connected in series with MIFFNet, both the segmentation accuracy and the background mis-segmentation probability were improved. In addition, in the ICC-2 organoid OCT data mixed with normal culture and drug action, the two-dimensional baseline method has lower segmentation accuracy than the pure three-dimensional network nnUNet 3D trained with sparsely labeled datasets; when the two-dimensional baseline network is connected in series with MIFFNet, the segmentation accuracy is close to or even exceeds that of nnUNet 3D, and the probability of background mis-segmentation decreases significantly. All three networks can be controlled at around 10%. It can be seen that the two-dimensional baseline network and MIFFNet in series can improve the consistency and stability of organoid segmentation before and after drug addition.

[0090] The segmentation accuracy and stability of the two-dimensional baseline method and MIFFNet in series are close to or even exceed those of the three-dimensional network, while the Infer-time and Memory indicators show that the computing time and video memory overhead are significantly reduced. Compared with the three-dimensional network, the video memory is reduced by about 50% and the computing time is shortened by more than 90%.

[0091] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transient storage media containing computer-usable program code.

[0092] The present invention can provide computer program instructions to a management platform of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the management platform of the computer or other programmable data processing device produce a device for implementing the system.

[0093] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions of the system.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions of the system.

Claims

1. An organoid image segmentation method, characterized in that: include: Generate graph nodes and neighboring graph nodes based on high-dimensional feature graphs of the current frame and neighboring frames, where the current frame is encoded as a target graph node, and context information of its three-dimensional adjacent area is encoded as a neighboring graph node; The spatial position constraint principle of the original three-dimensional image is used to dynamically generate the connection relationship between the graph nodes and the adjacent graph nodes, and a graph structure with physical structure consistency is constructed; The multi-dimensional information of neighboring graph nodes is transferred to the target graph node through graph convolution operation to compensate for the feature loss of the current frame, and the three-dimensional neighborhood information of neighboring frames is aggregated through the three-dimensional convolution kernel to achieve cross-dimensional inter-frame feature complementary fusion.

2. The method according to claim 1, characterized in that: The multi-dimensional inter-frame information fusion network is sequentially connected to the two-dimensional segmentation network as a preprocessing module, and the enhanced feature map output by the fusion network is input into the two-dimensional segmentation network.

3. The method according to claim 1, characterized in that: Multi-label annotation is performed on 3D OCT images to distinguish between organs, matrix gel and background areas, and the 3D continuity of the annotations is verified frame by frame. The training set is enhanced through random affine transformation, elastic deformation and combined intensity transformation.

4. The method according to claim 1, characterized in that: Multi-level feature maps of the current frame and adjacent frames are extracted through a lightweight two-dimensional convolutional network and stacked into a three-dimensional tensor. The neighborhood information of the three-dimensional tensor is aggregated using a zero-filled three-dimensional convolution operation to generate neighborhood graph nodes that are consistent with the physical space distribution.

5. The method according to claim 4, characterized in that: Based on the three-dimensional spatial position constraints, the adjacent nodes closely related to the target graph nodes are automatically screened, redundant connections are eliminated, the feature similarity between nodes is represented by the adjacency matrix, and the connection weights are dynamically adjusted.

6. The method according to claim 5, characterized in that: A multi-layer perceptron is used to fuse the features of the target graph node and its neighboring nodes. The update formula is: In the formula, is the neighboring node features aggregated by 3D convolution; is a multi-layer perceptron.

7. The method according to claim 1, characterized in that: Through step-by-step upsampling and skip connection operations, the fused feature map and the encoder multi-level features are channel-concatenated to restore the high-resolution segmentation result based on the U-Net architecture.

8. The method according to claim 1, characterized in that: The cascade network is suitable for the OCT image segmentation task of patient-derived organoids. It can dynamically adapt to the image contrast loss and noise enhancement caused by drug intervention, and balance the segmentation weights of organoids and background through a composite loss function.

9. The method according to claim 1, characterized in that: The model training adopts a composite loss function that combines Dice Loss and cross entropy loss, and performs end-to-end training through the Adam optimizer. The segmentation accuracy is quantified based on the Dice coefficient, background mis-segmentation rate, and inter-frame continuity indicators.

10. An organoid image segmentation system, characterized in that: The implementation of the system is based on the method according to any one of claims 1 to 9; The system comprises: Encoder, used to extract multi-level feature maps of the current frame and adjacent frames and convert them into three-dimensional tensors; Multi-dimensional inter-frame information fusion module, used to realize neighboring node information aggregation and feature update; The decoder is used to output the enhanced feature map of the current frame through upsampling and feature fusion, which is input into the two-dimensional segmentation network to complete organoid segmentation.

Images