Real-time data monitoring and automatic alarm method and system for brain surgery

Abstract

The invention discloses a brain surgery real-time data monitoring and automatic alarm method and system. The brain surgery real-time data monitoring and automatic alarm method comprises the steps that a multi-modal medical image and physiological parameter data are collected; processing the collected data by utilizing a visual language diffusion model; analyzing the processed data by using a panoramic segmentation architecture; performing lesion segmentation in real time; performing anomaly detection based on the segmentation result and the physiological parameter data; and when abnormity is detected, automatic alarm is triggered.

Description

Technical Field

[0001] This application relates to the field of medical technology, and particularly to a method and system for real-time data monitoring and automatic alarm in neurosurgery. Background Art

[0002] Neurosurgery is a complex and high-risk medical procedure that requires continuous and precise monitoring of the patient's physiological state. Traditional monitoring methods mainly rely on the experience and manual observation of medical staff, and have multiple defects.

[0003] First of all, manual monitoring is prone to fatigue and negligence, which may lead to missing key abnormal situations. Secondly, the ability to comprehensively analyze multi-modal data is limited, and it is difficult to detect potential risks in a timely manner. In addition, there is a lack of intelligent analysis of real-time medical images during the operation, and it is impossible to quickly identify changes in the lesion area. The alarm mechanism is not flexible enough, and there may be cases of missed alarms or false alarms. Finally, the efficiency of real-time processing and analysis of data is low, affecting the timeliness of decision-making. Summary of the Invention

[0004] In view of this, this application provides a method and system for real-time data monitoring and automatic alarm in neurosurgery, which solves the problems of insufficient intelligence, low real-time performance, and inflexible alarm mechanism in the existing neurosurgery monitoring technology.

[0005] An embodiment of this application provides a method for real-time data monitoring and automatic alarm in neurosurgery, including:

[0006] Collect multi-modal medical images and physiological parameter data;

[0007] Process the collected data using a vision-language diffusion model;

[0008] Analyze the processed data using a panoramic segmentation architecture;

[0009] Perform lesion segmentation in real time;

[0010] Perform anomaly detection based on the segmentation result and physiological parameter data;

[0011] When an anomaly is detected, trigger an automatic alarm.

[0012] The multi-modal medical images include real-time MRI images, real-time CT images, and optical coherence tomography (OCT) images.

[0013] The physiological parameter data includes heart rate, blood pressure, blood oxygen saturation, and electroencephalogram (EEG) data.

[0014] Before the vision-language diffusion model processes the collected data, it further includes:

[0015] Select a pre-trained StableDiffusion model as the base model;

[0016] Select BiomedCLIP as the medical-specific image and text encoder;

[0017] Fine-tune the model using a medical image dataset related to brain surgery;

[0018] Integrate the brain surgery-specific MAME diffusion model with the base model.

[0019] The processing of the collected data using the vision-language diffusion model includes:

[0020] Input the collected multi-modal medical images into the vision encoder to generate image feature representations;

[0021] Input the relevant medical text descriptions into the text encoder to generate text feature representations;

[0022] Fuse the image feature representations and text feature representations to generate multi-modal feature representations;

[0023] Use the diffusion model to iteratively optimize the multi-modal feature representations to generate enhanced feature representations;

[0024] Input the enhanced feature representations into the decoder to generate processed medical images or segmentation masks.

[0025] The panoramic segmentation architecture includes:

[0026] A multi-scale feature extraction module based on Transformer;

[0027] A self-attention mechanism to capture long-range dependencies;

[0028] A decoder module to generate fine-grained segmentation results;

[0029] A multi-modal feature fusion module.

[0030] The real-time lesion segmentation steps include:

[0031] Implement an efficient data reading and preprocessing module;

[0032] Design a data caching mechanism to reduce I / O overhead;

[0033] Use model quantization techniques to reduce computational complexity;

[0034] Adopt a TensorRT inference engine to accelerate model inference;

[0035] Implement model parallel computing to make full use of hardware resources.

[0036] The abnormal detection steps include:

[0037] Abnormal detection based on statistical methods;

[0038] Abnormal detection based on machine learning, including Isolation Forest and One-Class SVM;

[0039] Abnormal detection based on deep learning, including autoencoders and Generative Adversarial Network GAN;

[0040] Feature-level fusion, integrating image segmentation results and physiological parameter data;

[0041] Decision-level fusion, integrating the prediction results of multiple models.

[0042] The automatic alarm steps include:

[0043] Formulating initial alarm rules based on medical expert knowledge;

[0044] Implementing an adaptive alarm threshold, dynamically adjusted according to the surgical stage;

[0045] Developing a visual alarm interface to clearly display abnormal situations;

[0046] Implementing a multi-level alarm mechanism to issue alarms at different levels according to the degree of abnormality;

[0047] Designing an alarm log system to record and trace all alarm events.

[0048] The embodiment of the present application also provides a computer device, which includes:

[0049] At least one processor; and,

[0050] A memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for real-time data monitoring and automatic alarm of the above-mentioned brain surgery.

[0052] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions for causing a computer to execute the method for real-time data monitoring and automatic alarm of the above-mentioned brain surgery.

[0053] The embodiment of the present application also provides a computer program product, including computer instructions, characterized in that when the computer instructions are executed by a processor, the steps of the method for real-time data monitoring and automatic alarm of the above-mentioned brain surgery are implemented.

[0054] The present application has the following technical effects:

[0055] By collecting multi-modal medical images and physiological parameter data, the comprehensive monitoring of the brain surgery process is realized, improving the comprehensiveness and accuracy of the monitoring. The collected data is processed using a vision-language diffusion model, enhancing the efficiency and quality of data processing. A panoramic segmentation architecture is used to analyze the processed data, achieving precise segmentation and identification of the surgical area.

[0056] Lesion segmentation is performed in real time, enabling the timely detection of abnormal situations during the surgery. Abnormality detection is carried out based on the segmentation results and physiological parameter data, improving the accuracy and reliability of the abnormality detection. When an abnormality is detected, an automatic alarm is triggered, achieving a rapid response and reducing the risk of human delay.

[0057] Through an adaptive alarm threshold and a multi-level alarm mechanism, the flexibility and pertinence of the alarm are improved, reducing false alarms and missed alarms. The use of a variety of advanced technologies, such as vision-language diffusion models, panoramic segmentation architectures, deep learning abnormality detection, etc., greatly enhances the intelligent level and performance of the system. Brief Description of the Drawings

[0058] Figure 1 It is a flowchart of the real-time data monitoring and automatic alarm method for brain surgery provided by an embodiment of the present application;

[0059] Figure 2 It is a structural block diagram of the real-time data monitoring and automatic alarm system for brain surgery provided by an embodiment of the present application. Detailed Embodiments

[0060] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0061] It should be clear that the following illustrates the implementation manners of the present disclosure through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific implementation manners, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts belong to the scope of protection of the present disclosure.

[0062] It should be noted that the following description relates to various aspects of embodiments within the scope of the appended claims. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or practice a method. Additionally, this apparatus and / or method can be implemented using other structures and / or functionality in addition to one or more of the aspects set forth herein.

[0063] It should also be noted that the diagrams provided in the following embodiments merely illustrate the basic concept of the present disclosure schematically. Only the components related to the present disclosure are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0064] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the aspects described can be practiced without these specific details.

[0065] Embodiment 1

[0066] As Figure 1 shown, an embodiment of the present application provides a method for real-time data monitoring and automatic alarm in brain surgery, including the following steps:

[0067] S1: Collect multimodal medical images and physiological parameter data

[0068] During brain surgery, it is crucial to collect multimodal medical images and physiological parameter data in real time. The purpose of this step is to obtain comprehensive and accurate patient status information, providing a basis for subsequent analysis and decision-making.

[0069] Multimodal medical images include real-time MRI images, real-time CT images, and optical coherence tomography (OCT) images. These different types of images can provide complementary information, helping to more comprehensively understand the patient's brain condition. For example, MRI images can provide high-resolution soft tissue structure information, CT images can clearly show bone and calcified structures, while OCT images can provide tissue structure details at the micron level.

[0070] Physiological parameter data includes heart rate, blood pressure, blood oxygen saturation, and electroencephalogram (EEG) data. These parameters can reflect the overall physiological state and brain function activities of the patient. For example, changes in heart rate and blood pressure may indicate the stress state of the patient, blood oxygen saturation reflects the oxygen supply situation of brain tissue, and EEG data can monitor abnormalities in brain electrical activities.

[0071] During the data acquisition process, the following aspects need to be considered:

[0072] 1. Data acquisition frequency: Set an appropriate acquisition frequency according to the characteristics of different types of data. For example, EEG data may require a relatively high sampling rate (such as 250 Hz or higher), while the update frequency of MRI images may be relatively low.

[0073] 2. Data synchronization: Ensure that data from different modalities is synchronized in time, which is crucial for subsequent multi-modal fusion analysis. It can be achieved using a unified timestamp or dedicated synchronization hardware.

[0074] 3. Data quality control: Monitor the data quality in real-time, detect and mark possible artifacts or noises. For example, for MRI images, motion artifacts can be detected in real-time; for EEG data, electromyogram interference can be identified and filtered out.

[0075] 4. Data transmission and storage: Use a high-speed, low-latency network transmission protocol (such as 5G or a dedicated fiber optic network) to ensure real-time data transmission. At the same time, adopt an efficient data compression algorithm and a distributed storage system to handle a large amount of real-time data streams.

[0076] Through this step, the embodiments of the present invention can obtain comprehensive and real-time patient status information, laying a foundation for subsequent data processing and analysis. This multi-modal, multi-parameter monitoring method greatly improves the ability to perceive the patient's status, helping to detect potential risks and abnormal situations in a timely manner.

[0077] S2: Process the acquired data using a vision-language diffusion model

[0078] The vision-language diffusion model is an advanced deep learning technology that can effectively process and fuse multi-modal medical data. In the scenario of brain surgery, the main purpose of using this model is to improve the efficiency and quality of data processing, providing a more reliable basis for subsequent analysis and decision-making.

[0079] S2.1: Select a pre-trained StableDiffusion model as the base model

[0080] The StableDiffusion model is a generative model that generates high-quality images through a step-by-step denoising process. In medical image processing, selecting a pre-trained StableDiffusion model as the base has the following advantages:

[0081] 1. Strong generalization ability: The pre-trained model has learned rich feature representations on a large amount of data, which helps to process various complex medical images.

[0082] 2. High training efficiency: Using the pre-trained model can greatly reduce the time and computing resources required for training from scratch.

[0083] 3. Good stability: The stable diffusion model has good stability during the generation process, which helps to reduce artifacts and noise in medical image processing.

[0084] Specifically, it is possible to select mature pre-trained models such as Stable Diffusion v2.1 or DALL-E 2 as the starting point. These models are usually implemented using deep learning frameworks such as PyTorch or TensorFlow and can be directly loaded through the model libraries provided by these frameworks.

[0085] S2.2: Select BiomedCLIP as the medical-specific image and text encoder

[0086] BiomedCLIP is a multi-modal encoder designed specifically for the biomedical field, which can process medical images and related text descriptions simultaneously. The reasons for selecting BiomedCLIP include:

[0087] 1. Domain adaptability: BiomedCLIP has been pre-trained on a large amount of biomedical data and is more suitable for processing specific data in brain surgery.

[0088] 2. Multi-modal fusion: It can effectively fuse image and text information in a unified feature space, which is beneficial for subsequent analysis.

[0089] 3. Rich semantic information: By introducing text descriptions, more semantic information can be captured, improving the model's understanding ability.

[0090] Specifically, BiomedCLIP can be loaded through a dedicated biomedical AI library (such as BioMedIA). When using it, image data and the corresponding medical description text (such as radiology reports) need to be input into the model simultaneously.

[0091] S2.3: Fine-tune the model using a medical image dataset related to brain surgery

[0092] Although the pre-trained model has strong generalization ability, it is still necessary to fine-tune for specific tasks in brain surgery. The purpose of fine-tuning is:

[0093] 1. Adapt to specific tasks: Make the model better adapt to specific image features and patterns in neurosurgery.

[0094] 2. Improve accuracy: By learning the data distribution in a specific domain, improve the performance of the model on the target task.

[0095] 3. Accelerate convergence: Compared with training from scratch, fine-tuning can reach the ideal performance faster.

[0096] Specifically include:

[0097] Prepare the dataset: Collect a medical image dataset related to neurosurgery, including MRI, CT, and OCT images, as well as corresponding expert annotations.

[0098] Data augmentation: Use techniques such as rotation, scaling, and flipping to increase data diversity.

[0099] Fine-tuning strategy: Adopt a strategy of unfreezing layers one by one. First, fine-tune the top layer, and then gradually unfreeze the lower layers for fine-tuning.

[0100] Learning rate adjustment: Use a smaller learning rate (such as 1e-4 to 1e-5) for fine-tuning, and a learning rate decay strategy can be adopted.

[0101] Verification: Use cross-validation to ensure that the model does not overfit.

[0102] S2.4: Integrate the neurosurgery-specific MAME diffusion model with the base model

[0103] The MAME (Multimodal Attention and Masking Expansion) diffusion model is a diffusion model specifically designed for multimodal medical data. The purpose of integrating it with the base model is:

[0104] 1. Enhance multimodal processing ability: The MAME model can better process and fuse multimodal medical data.

[0105] 2. Improve specificity: Through integration, the model can better capture the specific features of neurosurgery.

[0106] 3. Improve the attention mechanism: The attention mechanism of the MAME model helps the model focus on key regions and features.

[0107] Specifically include:

[0108] Model fusion: Use knowledge distillation technology to transfer the knowledge of the MAME model to the base model.

[0109] Attention mechanism optimization: Implement a multi-head attention mechanism, allowing the model to simultaneously focus on features of different modalities and different scales.

[0110] Masking Strategy: Implement a dynamic masking strategy to better handle incomplete or noisy data.

[0111] S2.5: Input the collected multi-modal medical images into a visual encoder to generate image feature representations

[0112] The purpose of this step is to transform complex multi-modal medical images into feature representations that can be effectively processed by a computer. Visual encoders typically adopt deep convolutional neural network (CNN) architectures such as ResNet, DenseNet, or EfficientNet. These networks are pre-trained and can extract hierarchical features of the images.

[0113] In practical applications, embodiments of the present invention first preprocess the input medical images, including size adjustment, normalization, and data augmentation. Then, the processed images are input into the visual encoder. The shallow networks of the encoder extract low-level features such as edges and textures, while the deep networks capture more abstract high-level features. Finally, embodiments of the present invention obtain a high-dimensional feature vector or feature map that contains the key information of the original image.

[0114] To adapt to multi-modal images, embodiments of the present invention can design dedicated encoding paths. For example, for MRI images, embodiments of the present invention can use 3D convolutional networks; for temporal OCT data, CNN and LSTM can be combined. This can better capture the specific features of different modal images.

[0115] S2.6: Input relevant medical text descriptions into a text encoder to generate text feature representations

[0116] Text information, such as radiology reports or surgical records, contains rich semantic information that can complement image data. The role of the text encoder is to transform this text information into dense vectors so that they can be fused with image features.

[0117] Embodiments of the present invention typically use models based on the Transformer architecture as text encoders, such as BERT or its medical domain variant BioBERT. These models can effectively capture context information and long-distance dependencies in the text.

[0118] In practical applications, embodiments of the present invention first preprocess the input text, including word segmentation, stop word removal, etc. Then, the processed text is input into the encoder. The encoder generates representations for each token, and embodiments of the present invention can use the average of these token representations or the representation of the [CLS] token as the feature representation of the entire text.

[0119] To better adapt to the medical field, in the embodiments of the present invention, large-scale medical literature can be used to pre-train the model, or fine-tuning can be performed on a specific brain surgery text dataset. This can improve the model's ability to understand professional terms and specific expressions.

[0120] S2.7: Fuse the image feature representation and the text feature representation to generate a multimodal feature representation

[0121] Feature fusion is a key step in multimodal learning. Its goal is to organically combine information from different modalities to generate a unified and more informative representation.

[0122] Common fusion methods include simple concatenation, weighted summation, as well as more complex attention mechanisms and bilinear pooling. In this method, the embodiments of the present invention adopt a cross-modal attention mechanism based on Transformer. This mechanism allows the model to learn the correlations between different modalities and dynamically adjust the importance of each modality.

[0123] In specific implementation, the embodiments of the present invention first project the image features and text features into a space of the same dimension. Then, the multi-head attention mechanism is used to calculate the cross-modal attention weights. This process allows the model to focus on the image regions related to the text description, or the text parts related to the image content. Finally, the embodiments of the present invention fuse the attention-weighted features to obtain the final multimodal feature representation.

[0124] To improve the fusion effect, the embodiments of the present invention can introduce the idea of contrastive learning to encourage the model to learn more consistent cross-modal representations. In addition, the embodiments of the present invention can also design specific loss functions, such as maximizing mutual information, to further enhance the correlation between different modalities.

[0125] S2.8: Use a diffusion model to iteratively optimize the multimodal feature representation to generate an enhanced feature representation

[0126] The core idea of the diffusion model is to learn the data distribution by gradually adding and removing noise. In multimodal feature optimization, the embodiments of the present invention utilize this principle to enhance the quality and robustness of the feature representation.

[0127] In the implementation process, the embodiments of the present invention first define a noise schedule, which determines how to gradually increase the noise during the diffusion process and how to gradually remove the noise during the denoising process. Then, the embodiments of the present invention train a neural network to predict the noise at each step. This network usually adopts a U-Net architecture, which can effectively process features of different scales.

[0128] In the optimization stage, embodiments of the present invention start from the original multi-modal features and gradually add predefined noise. Then, embodiments of the present invention use the trained model to gradually remove the noise and generate enhanced feature representations. This process can help the model learn more robust and generalized feature representations, which is particularly effective when dealing with noisy or incomplete medical data.

[0129] To adapt to the characteristics of multi-modal data, embodiments of the present invention can incorporate conditional information, such as the patient's clinical indicators or surgical stage information, during the diffusion process. This can generate more targeted and personalized feature representations.

[0130] S2.9: Input the enhanced feature representation into the decoder to generate the processed medical image or segmentation mask

[0131] The role of the decoder is to convert the optimized feature representation back into the image space or generate a segmentation mask. The goal of this step is to generate high-quality, information-rich medical images or accurate lesion segmentation results.

[0132] For image generation tasks, embodiments of the present invention typically use transposed convolution (deconvolution) or a combination of upsampling + convolution to gradually increase the spatial resolution of the feature map. During this process, embodiments of the present invention can use skip connections to fuse features at different scales to retain more detailed information.

[0133] For segmentation tasks, embodiments of the present invention can use architectures such as fully convolutional networks (FCNs) or U-Nets. These architectures can generate segmentation masks of the same size as the input image, where each pixel is classified into a specific tissue type or lesion area.

[0134] To improve the quality of the generated results, embodiments of the present invention can introduce a combination of multiple loss functions. For example, for image generation, embodiments of the present invention can use a combination of pixel-level L1 or L2 loss, perceptual loss, and adversarial loss. For segmentation tasks, embodiments of the present invention can use Dice loss, cross-entropy loss, etc.

[0135] In addition, embodiments of the present invention can also introduce post-processing steps to further optimize the results. For example, for segmentation results, embodiments of the present invention can use conditional random fields (CRFs) to refine the boundaries; for generated images, embodiments of the present invention can apply super-resolution techniques to improve the image quality.

[0136] Through these steps, embodiments of the present invention can fully utilize the advantages of multi-modal data to generate high-quality medical images or accurate segmentation results, providing a reliable basis for subsequent analysis and decision-making.

[0137] Example: Real-time monitoring during brain tumor resection surgery

[0138] Suppose a real-time monitoring of a brain tumor resection surgery is being carried out in an embodiment of the present invention. The patient is a 50-year-old male with a glioma about 3 cm in diameter in the right frontal lobe. The key to the surgery is to accurately locate the tumor boundary and monitor the condition of the surrounding healthy brain tissue in real time during the resection process.

[0139] First, an embodiment of the present invention selects a pre-trained Stable Diffusion model as the base model (S2.1). This model has been trained on a large number of general medical images before and has powerful image generation and understanding capabilities. Then, an embodiment of the present invention selects BiomedCLIP as the medical-specific image and text encoder (S2.2). BiomedCLIP has been pre-trained on a large number of biomedical literature and images and is particularly good at understanding medical terms and image content.

[0140] To make the model better adapt to the specific requirements of brain surgery, an embodiment of the present invention fine-tunes the model using a dataset containing thousands of brain tumor MRI and CT images (S2.3). This dataset contains various types and sizes of brain tumors, as well as corresponding expert annotations. Through fine-tuning, the model learns to recognize the characteristics of different types of brain tumors and the boundaries between tumors and surrounding healthy tissues.

[0141] Subsequently, an embodiment of the present invention integrates the MAME (Multimodal Attention and Masking Expansion) diffusion model specific to brain surgery with the base model (S2.4). The MAME model is specifically designed to process multimodal data during the surgery, including real-time MRI, surgical microscope video streams, and various physiological parameters.

[0142] During the surgery, the system continuously acquires multimodal medical images. For example, intraoperative MRI images are acquired every 30 seconds, and at the same time, the video stream of the surgical microscope is continuously collected. These images are input into the visual encoder to generate image feature representations (S2.5). At the same time, the voice recognition system in the operating room transcribes the oral records of the surgeon in real time, such as "Approaching the tumor edge, slight bleeding observed". These texts are input into the text encoder to generate text feature representations (S2.6).

[0143] The system then fuses the image feature representation and the text feature representation to generate a multimodal feature representation (S2.7). This fusion process takes into account the visual information in the image and the semantic information described by the doctor, forming a comprehensive understanding of the current surgical state.

[0144] Next, the diffusion model iteratively optimizes this multimodal feature representation (S2.8). During this process, the model gradually removes the noise in the features and enhances the key information. For example, it may strengthen the features at the tumor margin while suppressing the irrelevant background information.

[0145] Finally, the optimized features are input into the decoder (S2.9). The decoder generates two outputs: one is the enhanced medical image, which highlights the tumor boundary and important anatomical structures; the other is the precise segmentation mask, which clearly marks the tumor region, the surrounding healthy brain tissue, and the functional areas that may be affected by the surgery.

[0146] This process is repeated every few seconds, providing the surgeon with continuously updated and highly accurate surgical navigation information. For example, if during the resection process, the model detects abnormal tissue changes near the tumor margin, it will immediately highlight this area in the enhanced image and update the segmentation mask to reflect this change.

[0147] At the same time, the system combines the doctor's verbal descriptions to understand and predict possible complications. For instance, if the doctor mentions "observed slight bleeding", the system will pay special attention to the image changes in the relevant area and may generate a predictive bleeding diffusion model to help the doctor evaluate the potential risks.

[0148] In this way, the vision-language diffusion model can provide real-time, accurate, and insightful information support during brain tumor surgery, greatly improving the safety and success rate of the surgery.

[0149] S3: Analyze the processed data using a panoptic segmentation architecture

[0150] Panoptic segmentation is an advanced computer vision technology that combines the advantages of semantic segmentation and instance segmentation, and can simultaneously identify object categories, object instances, and background regions in an image. In the context of brain surgery, the main purpose of using a panoptic segmentation architecture is to perform refined analysis on the processed medical images, accurately identify and locate various anatomical structures, lesion regions, and surgical-related instruments and equipment.

[0151] The panoptic segmentation architecture usually consists of several key components: a Transformer-based multi-scale feature extraction module, a self-attention mechanism, a decoder module, and a multimodal feature fusion module. These components work together to achieve high-precision image analysis.

[0152] The Transformer-based multi-scale feature extraction module is a core part of the panoramic segmentation architecture. It utilizes the powerful capabilities of Transformer to capture the long-range dependencies of images while retaining the advantages of CNN in local feature extraction. This module extracts features of different resolutions through a feature pyramid network (FPN) at multiple scales, enabling the model to simultaneously focus on large-scale context information and small-scale detail information.

[0153] In the implementation process, the embodiments of the present invention can use an architecture similar to Swin Transformer, which balances computational efficiency and performance through a self-attention mechanism with sliding windows. In addition, the embodiments of the present invention can also introduce deformable convolution to enhance the model's adaptability to irregular shapes, which is particularly important when dealing with complex brain structures.

[0154] The self-attention mechanism is another key component that allows the model to capture the interrelationships between pixels globally. In medical image analysis, this mechanism is particularly useful as it can help the model understand the spatial relationships and context information between different anatomical structures. The embodiments of the present invention can implement multi-head self-attention, enabling the model to simultaneously focus on different types of feature relationships. In addition, the embodiments of the present invention can also introduce position encoding to enhance the model's perception ability of spatial positions.

[0155] The role of the decoder module is to map the extracted feature maps back to the original image space to generate fine-grained segmentation results. The embodiments of the present invention can use a top-down path similar to FPN to gradually fuse features of different scales. At each decoding stage, the embodiments of the present invention can insert an attention module to better utilize the context information. In addition, the embodiments of the present invention can also introduce a boundary refinement module, such as Atrous Spatial Pyramid Pooling (ASPP), to improve the accuracy of the segmentation boundary.

[0156] The purpose of the multi-modal feature fusion module is to organically combine information from different modalities (such as MRI, CT, OCT). The embodiments of the present invention can use an attention mechanism to dynamically adjust the weights of different modalities, or use a graph convolutional network (GCN) to model the relationships between different modalities. In addition, the embodiments of the present invention can also introduce a loss function of mutual information maximization to encourage the model to learn more consistent cross-modal representations.

[0157] During the training process, the embodiments of the present invention need to design a loss function suitable for the panoramic segmentation task. This usually includes pixel-level cross-entropy loss, Dice loss to improve segmentation accuracy, and Focal loss to handle class imbalance problems. For the instance segmentation part, the embodiments of the present invention can introduce mask IoU loss to optimize the instance boundaries. In addition, the embodiments of the present invention can also design specific loss functions to encourage the model to learn anatomically reasonable segmentation results, such as by introducing shape priors or topological constraints.

[0158] To improve the robustness and generalization ability of the model, the embodiments of the present invention can adopt various data augmentation techniques, such as random cropping, rotation, scaling, brightness and contrast adjustment, etc. In addition, the embodiments of the present invention can also use advanced techniques such as MixUp or CutMix to further enhance the performance of the model.

[0159] In the inference stage, the embodiments of the present invention can use techniques such as Test Time Augmentation and model ensembling to further improve the segmentation accuracy. In addition, the embodiments of the present invention can also introduce post-processing steps, such as conditional random fields (CRF) or morphological operations, to refine the segmentation results.

[0160] By using this advanced panoramic segmentation architecture, the embodiments of the present invention can perform high-precision analysis of medical images in brain surgery, providing a reliable basis for subsequent anomaly detection and decision support. This method can not only accurately identify and locate various anatomical structures and lesion areas, but also distinguish different instances (such as multiple tumors or blood vessels), thus providing doctors with more comprehensive and detailed information support.

[0161] Example: Real-time image analysis of brain tumor surgery using a panoramic segmentation architecture

[0162] In the brain tumor resection surgery of the embodiments of the present invention, the panoramic segmentation architecture is used to perform real-time and precise analysis of the surgical area. This architecture can simultaneously process image data from multiple sources, including intraoperative MRI, surgical microscope video streams, and intraoperative ultrasound images.

[0163] First, the Transformer-based multi-scale feature extraction module starts to process the input image data. For example, for intraoperative MRI images, the system creates a feature pyramid. At the lowest level, the model focuses on fine texture and edge information, perhaps the subtle boundaries between the tumor and surrounding tissues. At higher levels, the model captures larger-scale structural information, such as the overall shape and location of the tumor. For the surgical microscope video stream, the model not only analyzes the current frame but also considers the information of several frames before and after to capture the dynamic changes of the surgical operation.

[0164] Suppose at a certain moment, the MRI image shows a tiny protrusion at the edge of the tumor, while the surgical microscope video shows a slight color change in this area. The multi-scale feature extraction module will capture these two details at the same time, providing important clues for subsequent analysis.

[0165] Next, the self-attention mechanism comes into play. It allows the model to establish pixel-level associations globally. In the example of an embodiment of the present invention, the self-attention mechanism may find that a protrusion at the edge of the tumor is potentially associated with an important blood vessel that is far away. The identification of such long-range dependencies is crucial for assessing surgical risk.

[0166] At the same time, the model also makes associations between different modalities. For example, it may notice that a certain area in the MRI image corresponds to the tissue response observed in the surgical microscope video. This cross-modal association helps the model form a more comprehensive understanding.

[0167] The decoder module then starts working, gradually mapping the extracted features back to the original image space. In this process, the model generates a series of fine segmentation results. First, there is a rough outline that roughly divides the image into several main parts such as tumor area, normal brain tissue, cerebrospinal fluid cavity, etc. Then, as the decoding process goes deeper, the segmentation becomes more and more refined.

[0168] In the final segmentation result, the embodiment of the present invention can not only see the accurately depicted tumor boundary, but also identify the surrounding key structures. For example, an important blood vessel close to the tumor is clearly marked, and its direction and diameter are accurately calculated. The adjacent functional areas, such as the language center, are also marked and their positional relationship with the tumor is clearly displayed.

[0169] The multimodal feature fusion module plays a key role in the whole process. It not only fuses the information of different imaging modalities, but also integrates other relevant data. For example, the patient's preoperative functional MRI results are superimposed on the real-time segmentation results, showing the precise location of language and motor functional areas. In addition, data from intraoperative neurophysiological monitoring are also integrated in real time, providing dynamic information about the functional integrity of specific brain areas.

[0170] During surgery, the panoramic segmentation architecture continuously updates its segmentation results. As the surgeon carefully removes the tumor tissue, the system is able to track the progress of the removal in real time. It not only updates the remaining outline of the tumor, but also identifies subtle changes in the surrounding tissue that may be affected by the surgical manipulation.

[0171] For example, if a small blood vessel is accidentally approached during the resection process, the system will immediately highlight this area in the segmentation result and may trigger a warning. At the same time, the system can also identify tissue changes caused by edema or minor bleeding, which may be difficult to detect under a traditional surgical microscope.

[0172] The results of panoramic segmentation are presented to the surgeon in an augmented reality manner superimposed on the field of view of the surgical microscope. The doctor can see a clear, color-coded overlay that shows residual tumors, key blood vessels, functional areas, and potential risk areas. This intuitive visualization greatly enhances the doctor's spatial perception ability and helps to make more accurate surgical decisions.

[0173] In addition, the system can automatically calculate some key metrics based on the segmentation results. For example, it can estimate the remaining volume of the tumor in real time, calculate the resection rate, and predict the possible locations of residual tumors. These quantitative information provides an objective assessment of the surgical process for the doctor.

[0174] In this way, the panoramic segmentation architecture provides unprecedented precision and real-time performance in brain tumor surgery. It not only greatly improves the safety and integrity of the surgery, but also provides strong technical support for personalized and precise surgical strategies.

[0175] S4: Perform lesion segmentation in real time

[0176] Real-time lesion segmentation is a key part of the brain surgery monitoring system. Its main purpose is to quickly and accurately identify and locate the lesion area during the surgery, providing immediate visual feedback and decision support for the surgeon. This step requires both ensuring the segmentation accuracy and meeting the real-time requirement, which poses high demands on the efficiency of the algorithm and the performance of the hardware.

[0177] To achieve efficient real-time lesion segmentation, the embodiments of the present invention need to be optimized from multiple aspects:

[0178] First, at the data processing level, the embodiments of the present invention implement an efficient data reading and preprocessing module. This module uses multi-threading technology to read medical image data of different modalities in parallel, and at the same time performs data preprocessing on the GPU, such as image normalization, resampling, etc. The embodiments of the present invention also design a data caching mechanism to store frequently accessed data in memory or GPU memory to reduce I / O overhead. This method can significantly reduce the time for data loading and preprocessing, providing more time budget for real-time segmentation.

[0179] In terms of model design, the embodiments of the present invention adopt a lightweight network architecture, such as MobileNetV3 or EfficientNet-Lite as the backbone network. While maintaining relatively high accuracy, these networks significantly reduce the computational complexity and the number of parameters. The embodiments of the present invention also use depthwise separable convolutions to replace standard convolutions, further reducing the computational amount. In addition, the embodiments of the present invention introduce attention mechanisms, such as spatial attention and channel attention, to help the model more effectively focus on key regions and improve the segmentation accuracy.

[0180] To further improve the inference speed, the embodiments of the present invention adopt model quantization technology. By quantizing the weights and activation values of the model from 32-bit floating-point numbers to 8-bit integers, the embodiments of the present invention can significantly reduce the memory footprint and computational amount of the model, while having a relatively small impact on the accuracy. The embodiments of the present invention use a method that combines post-training quantization and quantization-aware training to achieve a good balance between speed and accuracy.

[0181] In the selection of the inference engine, the embodiments of the present invention adopt the TensorRT inference engine. TensorRT can automatically optimize the model, such as operator fusion, kernel auto-tuning, etc., and make full use of the computing power of the GPU. The embodiments of the present invention also use the dynamic shape feature of TensorRT, enabling the model to adapt to inputs of different sizes and enhancing the flexibility of the system.

[0182] To make full use of the hardware resources, the embodiments of the present invention implement model parallel computing. The embodiments of the present invention decompose the entire segmentation task into multiple subtasks, such as the segmentation of different anatomical structures, and then execute these subtasks in parallel on multiple GPUs. The embodiments of the present invention use the NVIDIA NCCL library to optimize the communication between GPUs, ensuring efficient data transmission and synchronization.

[0183] In terms of the segmentation algorithm, the embodiments of the present invention adopt a cascaded segmentation strategy. First, a fast but relatively rough model is used for preliminary segmentation, and then a more refined model is used to refine the region of interest. This method can reduce unnecessary calculations while ensuring accuracy.

[0184] The embodiments of the present invention also implement a dynamic scheduling system that dynamically adjusts the segmentation frequency and accuracy according to the current system load and surgical stage. For example, during critical surgical stages, the system will increase the segmentation frequency and accuracy; while during relatively stable stages, the frequency can be appropriately reduced to save computing resources.

[0185] To process real-time data streams, an embodiment of the present invention designs a pipeline processing system. This system includes multiple stages such as data preprocessing, model inference, and postprocessing. Each stage is executed in an independent thread or process, and data transfer is achieved through an efficient queue mechanism, realizing the parallelization of the entire processing flow.

[0186] Finally, an embodiment of the present invention implements a result visualization module, which overlays the segmentation result on the original image in real time and renders it to a display device through a high-performance graphics API (such as OpenGL or Vulkan). An embodiment of the present invention also implements an interactive 3D visualization function, allowing doctors to view the segmentation result from different angles and scales.

[0187] Through these optimization measures, the system of an embodiment of the present invention can achieve millisecond-level lesion segmentation while ensuring high precision, meeting the strict requirements of real-time monitoring in brain surgery. This real-time lesion segmentation technology provides timely and accurate visual feedback for surgeons, helping to improve the accuracy and safety of surgery.

[0188] Of course, I would be happy to provide a specific example for step S4. An embodiment of the present invention will continue to detail how real-time lesion segmentation is achieved in the context of a brain tumor resection surgery.

[0189] Example: Real-time Lesion Segmentation in Brain Tumor Surgery

[0190] Imagine that an embodiment of the present invention is performing a complex glioma resection surgery. The operating room is equipped with state-of-the-art equipment, including intraoperative MRI, high-resolution surgical microscopes, and ultrasound probes. The task of an embodiment of the present invention is to accurately segment the tumor and surrounding tissues in real time during the surgery, providing timely and accurate visual guidance for the surgeon.

[0191] First, an embodiment of the present invention implements an efficient data reading and preprocessing module. When the MRI scanner generates a new set of images every 30 seconds, the system of an embodiment of the present invention immediately starts processing. The data is transmitted to a computing server equipped with multiple GPUs through a high-speed fiber optic network. An embodiment of the present invention uses NVIDIA GPUDirect technology, which allows data to be directly transferred from the network interface card to the GPU memory, bypassing the CPU and significantly reducing the data transfer time.

[0192] The preprocessing stage adopts multi-threaded parallel processing. For example, when processing MRI images, one thread is responsible for geometric correction of the images, and another thread simultaneously performs intensity normalization. For the video stream of the surgical microscope, an embodiment of the present invention uses GPU-accelerated image enhancement algorithms, such as real-time denoising and contrast enhancement, which are directly performed on the GPU without transferring the data back to the CPU.

[0193] An embodiment of the present invention designs an intelligent data caching mechanism. The system will predict the next data block that may need to be processed and load it into the GPU memory in advance. For example, if the surgeon is approaching a certain area of the tumor, the system will preferentially cache the high-resolution data around that area.

[0194] In terms of model design, an embodiment of the present invention adopts a lightweight but efficient network architecture. The basic network uses MobileNetV3, which achieves a good balance between accuracy and computational efficiency. An embodiment of the present invention also introduces spatial and channel attention modules to help the model better focus on key areas, such as tumor boundaries and important anatomical structures.

[0195] To further improve the inference speed, an embodiment of the present invention applies model quantization technology. By quantizing the weights and activation values of the model from 32-bit floating-point numbers to 8-bit integers, an embodiment of the present invention reduces the model size by 75%, while increasing the inference speed by 3 times. An embodiment of the present invention uses quantization-aware training to ensure that the quantized model maintains high accuracy. In the tests of an embodiment of the present invention, the accuracy of the quantized model in the tumor boundary detection task drops by less than 0.5%, but the inference time is reduced from the original 100 milliseconds to 30 milliseconds.

[0196] An embodiment of the present invention uses the TensorRT inference engine to further optimize the model performance. TensorRT automatically performs a series of optimizations, including layer fusion, kernel auto-tuning, and dynamic tensor memory. For example, it fuses multiple small convolution operations into one large operation, reducing the number of memory accesses. In the brain tumor segmentation model of an embodiment of the present invention, these optimizations reduce the inference time by another 40% to reach 18 milliseconds.

[0197] To make full use of hardware resources, an embodiment of the present invention implements model parallel computing. The system of an embodiment of the present invention is equipped with 4 high-performance GPUs. The main segmentation task is decomposed into multiple subtasks and executed in parallel on different GPUs. For example, one GPU is responsible for processing the axial slices of the MRI image, another for processing the coronal slices, the third for processing the sagittal slices, and the fourth GPU is dedicated to processing the video stream of the surgical microscope. An embodiment of the present invention uses the NVIDIA NCCL library to optimize the communication between GPUs to ensure fast and efficient data synchronization.

[0198] An embodiment of the present invention also implements a dynamic load balancing system. If it is detected that the load of a certain GPU is particularly high, the system will automatically reallocate some tasks to the GPU with a lighter load. For example, at a certain stage of the operation, if the doctor mainly focuses on a specific area of the tumor, the system will allocate more computing resources to the GPU that processes the images of that area.

[0199] In actual operation, when a surgeon starts to remove a tumor, the system of the embodiment of the present invention can segment and analyze images in real time at a speed of 30 frames per second. For example, when the surgeon's surgical instrument approaches a suspicious area, the system immediately performs high-precision segmentation on that area. If healthy tissue is detected, the system will mark the area with a red contour on the augmented reality display, and at the same time update the three-dimensional reconstruction model in less than 100 milliseconds.

[0200] The system of the embodiment of the present invention also implements an adaptive segmentation strategy. During critical stages of the surgery, such as when approaching important blood vessels or functional areas, the system will automatically increase the segmentation accuracy, sacrificing some speed in exchange for higher accuracy. For example, the system may switch to a more complex but more accurate deep learning model, or increase the number of sampling points.

[0201] Finally, the embodiment of the present invention develops a real-time visualization module that directly overlays the segmentation results onto the field of view of the surgical microscope. Using OpenGL acceleration, the embodiment of the present invention can render complex 3D models with extremely low latency (less than 5 milliseconds). The surgeon can see a color-coded translucent overlay that clearly shows the location of the tumor boundary, key blood vessels, and functional areas.

[0202] Through this highly optimized and parallelized real-time lesion segmentation system, the embodiment of the present invention can provide millisecond-level responses during complex brain tumor surgeries. The system can not only accurately track changes in the tumor boundary but also promptly identify possible complications, such as small hemorrhages or unexpected tissue deformations. This greatly enhances the surgeon's operating precision and patient safety, making it possible to maximize tumor resection while preserving key functions.

[0203] S5: Perform anomaly detection based on the segmentation results and physiological parameter data

[0204] In a brain surgery monitoring system, anomaly detection is a crucial step. Its main purpose is to promptly detect possible problems during the surgery, such as unexpected bleeding, tissue deformation, abnormal physiological parameters, etc., so as to provide timely warnings to the medical team. This step combines the lesion segmentation results obtained in the previous steps and the real-time collected physiological parameter data, and realizes comprehensive and accurate anomaly detection through a variety of advanced algorithms.

[0205] The anomaly detection system of the embodiment of the present invention adopts a multi-level and multi-modal method, comprehensively using statistical methods, machine learning, and deep learning technologies to improve the accuracy and robustness of detection.

[0206] First, at the level of statistical methods, the embodiments of the present invention implement anomaly detection based on threshold and trend analysis. For various physiological parameters, such as heart rate, blood pressure, blood oxygen saturation, etc., the embodiments of the present invention set normal range thresholds based on medical knowledge. The system will monitor in real time whether these parameters exceed the thresholds and analyze their change trends. The embodiments of the present invention use time series analysis techniques such as Exponentially Weighted Moving Average (EWMA) to smooth short-term fluctuations and can respond quickly to significant changes. In addition, the embodiments of the present invention also implement anomaly detection based on Z-score, which can adaptively detect abnormal fluctuations relative to the patient's personal baseline.

[0207] At the level of machine learning, the embodiments of the present invention adopt a variety of unsupervised and semi-supervised learning algorithms. The Isolation Forest algorithm is used to detect anomaly points in physiological parameters, and it is particularly good at discovering rare outliers that are different from most normal data. The embodiments of the present invention also use One-class Support Vector Machine (One-class SVM), which identifies anomalies by learning the distribution of normal data. To process high-dimensional data, the embodiments of the present invention first use Principal Component Analysis (PCA) for dimensionality reduction and then apply these algorithms in the reduced-dimensional space, which can improve the efficiency and effectiveness of the algorithms.

[0208] Deep learning methods perform excellently in processing complex medical images and time series data. The embodiments of the present invention use an autoencoder-based anomaly detection method, training an autoencoder to reconstruct normal medical images and physiological parameter sequences, and then using the reconstruction error to identify anomalies. For medical images, the embodiments of the present invention also implement anomaly detection based on Generative Adversarial Network (GAN). The embodiments of the present invention train a GAN to generate normal brain images and then use the output of the discriminator and the differences between the generated images and the real images to detect anomalies.

[0209] To make full use of multi-modal data, the embodiments of the present invention implement feature-level fusion and decision-level fusion. In feature-level fusion, the embodiments of the present invention map the image segmentation results and physiological parameter data to a common feature space and then perform anomaly detection in this fused feature space. The embodiments of the present invention use an attention mechanism to dynamically adjust the importance of different modal features. In decision-level fusion, the embodiments of the present invention independently perform anomaly detection on each modality and then synthesize the prediction results of multiple models through weighted voting or learned fusion rules.

[0210] Considering the dynamics of the surgery, the embodiments of the present invention implement an online learning mechanism, enabling the model to continuously update and adapt during the surgery. This includes using the sliding window technique to update the parameters of the statistical model, and using incremental learning algorithms such as Online Random Forest to update the machine learning model.

[0211] To handle the temporal correlation of the data, the embodiments of the present invention introduce Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) to model the time series of physiological parameters. These models can capture long-term dependencies and help identify slowly developing abnormal patterns.

[0212] The embodiments of the present invention also pay special attention to the problem of handling data imbalance, because in actual surgeries, abnormal situations are usually rare. The embodiments of the present invention adopt a comprehensive oversampling and undersampling technique (SMOTEENN) to balance the training data, and use special loss functions such as focal loss to enhance the model's learning of the minority class.

[0213] At the system implementation level, the embodiments of the present invention adopt a distributed computing framework, such as Apache Spark, to process large-scale real-time data streams. The embodiments of the present invention also implement a hot update mechanism for the model, allowing the anomaly detection model to be updated without interrupting the system operation.

[0214] Finally, to improve the interpretability of the system, the embodiments of the present invention implement feature importance analysis based on SHAP (SHapley Additive exPlanations) values. This can help doctors understand which factors have an important impact on the anomaly detection results, so as to make more informed decisions.

[0215] Through this multi-level and multi-modal anomaly detection method, the system of the embodiments of the present invention can comprehensively and accurately identify various abnormal situations during the surgery, provide timely and reliable warning information for the medical team, thereby improving the safety and success rate of the surgery.

[0216] S6: When an anomaly is detected, trigger an automatic alarm

[0217] The automatic alarm system is the last line of defense in the real-time monitoring method for the entire neurosurgery, and it is also the key link to transform the analysis results of all the previous steps into actual actions. Its main purpose is to timely and accurately send an alarm to the medical team when an abnormal situation is detected, and at the same time provide clear and useful information support to help doctors make correct decisions quickly.

[0218] The automatic alarm system of the embodiment of the present invention adopts a multi-level, intelligent design to ensure the timeliness, accuracy and practicality of the alarm.

[0219] First, based on the knowledge of medical experts, the embodiment of the present invention has formulated a set of initial alarm rules. These rules cover various possible abnormal situations, including abnormal physiological parameters, abnormal changes in images, and compound abnormalities of multiple indicators. Each abnormal situation is assigned a corresponding severity level, which determines the urgency and method of the alarm.

[0220] However, fixed alarm rules may not be flexible enough to adapt to the specific conditions of each patient and the different stages of surgery. Therefore, the embodiments of the present invention implement an adaptive alarm threshold mechanism. The system dynamically adjusts the alarm threshold based on the patient's baseline data, the progress of the operation, and the previous alarm history. For example, during the critical stage of the operation, the system may lower the alarm threshold of certain parameters to increase vigilance; during the recovery period, the threshold may be appropriately relaxed to reduce unnecessary alarms.

[0221] To achieve this adaptive mechanism, the embodiment of the present invention uses an online learning algorithm. The system continuously monitors the doctor's response to the alarm, such as whether the alarm is confirmed, what action is taken, etc., and then uses this information to adjust the alarm model. The embodiment of the present invention adopts the idea of ​​reinforcement learning, using the doctor's positive feedback as a reward and negative feedback (such as frequently ignoring a certain type of alarm) as a penalty, thereby continuously optimizing the alarm strategy.

[0222] In terms of alarm triggering, the embodiments of the present invention implement a multi-level alarm mechanism. Depending on the severity and urgency of the detected abnormality, the system will select different alarm methods. This may include visual warnings on the display, audible alarms, notifications to the doctor's mobile device, and even direct notifications to the entire medical team in extreme cases. The embodiments of the present invention also implement an alarm escalation mechanism. If the primary alarm is not handled in a timely manner, the system will automatically escalate the alarm level.

[0223] In order to provide more contextual information, the embodiment of the present invention develops a visual alarm interface. This interface not only displays the specific reason for triggering the alarm, but also provides relevant historical data trends, image comparison and other information. The embodiment of the present invention uses data visualization technology, such as heat maps, trend charts, etc., to intuitively display abnormal situations. In addition, the interface also integrates artificial intelligence-assisted diagnosis functions, which can provide doctors with possible cause analysis and treatment suggestions.

[0224] Considering the particularity of the surgical environment, the alarm system of the embodiment of the present invention also supports gesture control and voice commands. The doctor can confirm the alarm, request more information or adjust the alarm settings through simple gestures or voice commands without interrupting the surgical operation.

[0225] To avoid "alarm fatigue", the embodiments of the present invention implement an intelligent alarm filtering and aggregation mechanism. The system analyzes the frequency and pattern of alarms, automatically identifies and suppresses duplicate or unnecessary alarms. For multiple related alarms, the system performs intelligent aggregation and presents them to the doctor in a more concise and informative manner.

[0226] The embodiments of the present invention also implement a comprehensive alarm log system. Each alarm event is detailedly recorded, including the trigger time, cause, severity, doctor's response, etc. These logs are not only used for post-event analysis and system improvement, but also provide valuable reference information for doctors during the operation.

[0227] At the system implementation level, the embodiments of the present invention adopt a highly available design, including mechanisms such as redundant servers and automatic failover, to ensure that the alarm system can work properly under any circumstances. The embodiments of the present invention also implement end-to-end encryption and access control to protect sensitive medical data.

[0228] Finally, to continuously improve the system performance, the embodiments of the present invention implement a feedback loop mechanism. After the operation, the system generates a detailed report, including all alarm events, doctor's responses, and operation results. The medical team can review this report, evaluate the accuracy and usefulness of the alarms, and provide feedback. This feedback is used to further optimize the alarm rules and models.

[0229] Through this intelligent and personalized automatic alarm system, the embodiments of the present invention can minimize the interference to doctors while ensuring timeliness and accuracy, providing strong safety guarantees for brain surgery.

[0231] Embodiment 2

[0232] Figure 2 FIG. is a schematic structural diagram of a real-time data monitoring and automatic alarm system for brain surgery provided by an embodiment of the present disclosure. The real-time data monitoring and automatic alarm system 200 for brain surgery includes:

[0233] A multimodal data acquisition module 21 for acquiring multimodal medical images and physiological parameter data;

[0234] A visual language diffusion model processing module 22 for processing the acquired data;

[0235] A panoramic segmentation module 23 for analyzing the processed data;

[0236] A real-time lesion segmentation module 24;

[0237] Anomaly detection module 25, configured to perform anomaly detection based on the segmentation result and physiological parameter data;

[0238] Automatic alarm module 26, configured to trigger an alarm when an anomaly is detected;

[0239] And a central processing unit 27, configured to coordinate and control the operation of the above-mentioned modules.

[0240] The system according to the embodiments of the present disclosure can execute the method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the system according to the embodiments of the present disclosure correspond to the steps in the method according to the embodiments of the present disclosure. For the detailed function descriptions of the modules in the system, reference may be specifically made to the descriptions in the corresponding methods shown above, and details are not repeated here.

[0241] The above are only optional implementation manners of some implementation scenarios of the present disclosure. It should be noted that for those of ordinary skill in the art, without departing from the technical concept of the solution of the present disclosure, other similar implementation means based on the technical idea of the present disclosure also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A method for real-time data monitoring and automatic alarming of brain surgery, characterized in that: The following steps are involved: Collect multimodal medical images and physiological parameter data; Use the visual language diffusion model to process the collected data; Analyze the processed data using a panoptic segmentation architecture; Perform lesion segmentation in real time; Perform anomaly detection based on segmentation results and physiological parameter data; When an abnormality is detected, an automatic alarm is triggered.

2. The method according to claim 1, characterized in that The multimodal medical images include real-time MRI images, real-time CT images and optical coherence tomography (OCT) images.

3. The method according to claim 1, characterized in that The physiological parameter data include heart rate, blood pressure, blood oxygen saturation and electroencephalogram (EEG) data.

4. The method according to claim 1, characterized in that: Before the visual language diffusion model processes the collected data, it also includes: Select the pre-trained stable diffusion model as the base model; Select BiomedCLIP as a medical-specific image and text encoder; Fine-tune the model using a medical image dataset related to brain surgery; Integrating the brain surgery-specific MAME diffusion model with the base model.

5. The method according to claim 4, characterized in that The processing of the collected data using the visual language diffusion model includes: Input the acquired multimodal medical images into the visual encoder to generate image feature representation; Input relevant medical text descriptions into the text encoder to generate text feature representations; Fusion of image feature representation and text feature representation to generate multimodal feature representation; Iteratively optimize the multimodal feature representation using a diffusion model to generate enhanced feature representation; The enhanced feature representation is fed into the decoder to generate a processed medical image or segmentation mask.

6. The method according to claim 1, characterized in that The panoptic segmentation architecture includes: Transformer-based multi-scale feature extraction module; Self-attention mechanism to capture long-range dependencies; Decoder module to generate refined segmentation results; Multimodal feature fusion module.

7. The method according to claim 1, characterized in that The real-time lesion segmentation step comprises: Implement efficient data reading and preprocessing modules; Design data caching mechanism to reduce I / O overhead; Use model quantization techniques to reduce computational complexity; Use TensorRT inference engine to accelerate model inference; Implement model parallel computing and make full use of hardware resources.

8. The method according to claim 1, characterized in that The anomaly detection step comprises: Anomaly detection based on statistical methods; Machine learning-based anomaly detection, including isolation forest and one-class SVM; Anomaly detection based on deep learning, including autoencoders and generative adversarial networks (GANs); Feature-level fusion, integrating image segmentation results and physiological parameter data; Decision-level fusion combines the prediction results of multiple models.

9. The method according to claim 1, characterized in that: The automatic alarm step comprises: Develop initial alarm rules based on medical expert knowledge; Implement adaptive alarm thresholds that are dynamically adjusted according to the surgical stage; Develop a visual alarm interface to clearly display abnormal situations; Implement a multi-level alarm mechanism to issue different levels of alarms according to the degree of abnormality; Design an alarm log system to record and trace back all alarm events.

10. A system for real-time data monitoring and automatic alarm in brain surgery, characterized in that: include: A multimodal data acquisition module, used for acquiring multimodal medical images and physiological parameter data; Visual language diffusion model processing module, used to process the collected data; Panoptic segmentation module, used to analyze the processed data; Real-time lesion segmentation module; Anomaly detection module, used for anomaly detection based on segmentation results and physiological parameter data; Automatic alarm module, used to trigger an alarm when an abnormality is detected; And a central processing unit, which is used to coordinate and control the operation of the above modules.

Images