Nucleus labelling apparatus and method, wearable xr device, related apparatus

Abstract

The present application provides a nucleus labeling device and method, a wearable XR device, and related devices. The method includes: obtaining medical imaging data of a patient's brain, and reconstructing a three-dimensional model of the patient's brain based on the medical imaging data; segmenting one or more nuclei from the medical imaging data to obtain a segmentation result for each nucleus; obtaining a first display image based on the segmentation result of each nucleus and the three-dimensional model, and using a wearable XR device to display the first display image. In the first display image, each nucleus is displayed separately, and the position and size of each nucleus is marked for the doctor wearing the XR device. The present application aims to provide a device and a wearable XR device that can accurately, quickly, and visually label and locate nuclei, so as to solve the problem of medical image processing and assist doctors in making decisions and operations.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep brain electrical stimulation, XR devices, computer vision, and deep learning, and in particular to a nucleus labeling device, a wearable XR device, a nucleus labeling method, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] With the development of technology and social progress, in the medical field, nucleus labeling aims to help doctors accurately locate and label the nucleus structure in the brain. The nucleus is a collection of specific regions in the brain, which is crucial for cognitive, motor control, and emotional regulation. Accurate labeling of the nucleus can help doctors better understand the brain structure of patients in surgical planning, neurologic disease diagnosis and treatment. At the same time, with the development of virtual reality (VR) and augmented reality (AR) technology, wearable XR devices such as AR glasses have become a powerful tool for medical image visualization and navigation. By superimposing computer-generated images on real-world visual scenes, XR devices can provide an intuitive interface between doctors and patients' brains, making medical operations more precise and safe.

[0003] Currently, nucleus labeling and XR device applications are relatively new technologies and are still in the development and improvement stage. In some areas or medical institutions, these technologies may not have been widely adopted or are limited to specific research institutions or high-end medical centers. Based on this, the present application provides a nucleus labeling device, a wearable XR device, a nucleus labeling method, a computer readable storage medium, and a computer program product to improve the existing technology. SUMMARY

[0004] The present application aims to provide a nucleus labeling device, a wearable XR device, a nucleus labeling method, a computer readable storage medium, and a computer program product to solve the problem of medical image processing and assist doctors in decision-making and operation.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a nucleus labeling device, which comprises a memory and at least one processor, the memory storing a computer program, and the at least one processor being configured to execute the computer program to implement the following steps:

[0007] Obtaining medical image data of a patient's brain, and reconstructing a three-dimensional model of the patient's brain according to the medical image data;

[0008] Segmenting one or more nuclei from the medical image data to obtain a segmentation result of each nucleus;

[0009] According to the segmentation result of each nucleus and the three-dimensional model, a first display image is obtained, and the first display image is displayed using a wearable XR device, in which each nucleus is displayed separately, and the position and size of each nucleus are marked for a doctor wearing the XR device.

[0010] The technical scheme has the beneficial effects that by obtaining medical image data of the patient's brain, and performing three-dimensional model reconstruction and nucleus segmentation, each nucleus can be accurately located and segmented, providing a high-precision marking result for the doctor; by using a wearable XR device, the position and size of each nucleus can be displayed in real time in the first display image, providing an intuitive marking experience for the doctor wearing the XR device, and the doctor can directly observe and mark the nucleus, improving the convenience and accuracy of operation. In summary, the nucleus marking device provides an accurate, real-time and convenient nucleus marking experience for the doctor by combining three-dimensional model reconstruction, nucleus segmentation and the use of a wearable XR device, which helps to improve the efficiency and accuracy of medical work, and promotes the progress of medical research and clinical practice.

[0011] In some possible implementation manners, the plurality of nuclei include one or more of the following: the nucleus accumbens, the anterior limb of the internal capsule, the subthalamic nucleus, the ventral intermediate nucleus of the thalamus, the medial globus pallidus, the ventral internal capsule, the ventral striatum and the superior lateral branch of the medial forebrain bundle.

[0012] The technical scheme has the beneficial effects that by using the nucleus marking device, a plurality of nuclei can be accurately marked and located, which helps the doctor to more accurately understand the brain structure and information; by segmenting and reconstructing the three-dimensional model of the patient's brain, combined with the segmentation result of the nuclei, more comprehensive and accurate medical image processing can be provided, which helps the doctor to better understand the patient's brain condition and provides a more reliable basis, so that the doctor can more accurately locate the target area. In summary, by providing accurate nucleus marking and visualization tools, the accuracy and efficiency of medical image processing and neurosurgery are improved, which helps to improve the success rate of surgery and the treatment effect of patients.

[0013] In some possible implementation manners, the XR device adopts an AR device, and the at least one processor is configured to obtain the first display image in the following manner when executing the computer program:

[0014] An real-time image is collected by a camera of the AR device, and the real-time image contains the patient's brain and one or more electrode leads implanted in the patient's brain;

[0015] The real-time image and the three-dimensional model are registered to obtain a registration matrix;

[0016] According to the segmentation result of each nucleus, the registration matrix and the real-time image, the first display image is obtained.

[0017] The technical scheme has the beneficial effects that: the real-time image of the patient's brain and the implanted electrode lead wire is collected by the camera of the AR device, and is registered with the three-dimensional model reconstructed in advance, so that the accurate image of the patient's brain can be obtained in real time, and is aligned with the model, thereby providing accurate basic data for subsequent processing and display; by combining the segmentation result and the registration matrix with the real-time image, the first display image can be generated, in which each nucleus is clearly displayed, and its position and size are labeled, and the doctor can intelligently navigate and locate the nucleus on the real-time image, better understand the patient's brain structure, and provide accurate guidance. In summary, by using the AR device as the XR device, and combining real-time image collection, registration and nucleus segmentation result, the doctor is provided with the functions of intelligent navigation and augmented reality auxiliary display, which helps to improve the accuracy, individualization and efficiency of the operation.

[0018] In some possible implementation manners, the at least one processor, when configured to execute the computer program, also implements the following steps:

[0019] In the process of implanting the electrode lead wire, according to the real-time image, real-time pose information of a target electrode lead wire currently implanted is obtained;

[0020] According to the real-time pose information, it is detected whether the target electrode lead wire deviates from a preset implantation path corresponding to the target electrode lead wire;

[0021] If deviated, a second display image is obtained according to the segmentation result of a target target point corresponding to the target electrode lead wire, the registration matrix, the real-time image, position information of the target electrode lead wire and the corresponding preset implantation path, and the AR device is used to display the second display image to assist the doctor to implant the target electrode lead wire according to the preset implantation path, in the second display image, the target electrode lead wire, the target target point corresponding to the target electrode lead wire and the preset implantation path are highlighted, and the target target point is one of the nuclei.

[0022] The technical scheme has the beneficial effects that: by using the real-time image, the current position and posture information of the electrode lead being implanted can be obtained, so that the doctor can know the position and direction of the lead in real time, thereby performing accurate operation and adjustment; by comparing the real-time pose information of the target electrode lead with the preset implantation path, whether the lead deviates from the expected position can be detected, which helps the doctor to discover and correct the deviation or error that may occur in the lead implantation process in time; according to the segmentation result of the target target point corresponding to the target lead, the registration matrix, the real-time image and the lead position information, the second display image is generated, which can provide more intuitive navigation information to help the doctor implant the target electrode lead according to the preset implantation path. In summary, the technical scheme provides real-time navigation and assistance, so that the doctor can implant the target electrode lead more accurately, improve the implantation precision, reduce the operation risk and ensure the achievement of the treatment effect.

[0023] In some possible implementation manners, the at least one processor, when configured to execute the computer program, also implements the following steps:

[0024] After the one or more electrode leads are implanted, a stimulation strategy corresponding to the patient is obtained, the stimulation strategy including a set of stimulation parameters corresponding to each of the electrode leads;

[0025] According to the set of stimulation parameters corresponding to each of the electrode leads, a stimulation result of each of the electrode leads is obtained, the stimulation result being used to indicate a stimulation area and a stimulation intensity of each area point in the stimulation area;

[0026] According to the segmentation result of each of the nuclei, the registration matrix, the real-time image and the stimulation result of each of the electrode leads, a third display image is obtained, and the third display image is displayed by using the AR device, in which the stimulation result of each of the nuclei and each of the electrode leads is visually displayed.

[0027] The beneficial effects of the technical solution are that: by obtaining the stimulation strategy corresponding to the patient, including the stimulation parameter set corresponding to each electrode lead, the individualized stimulation scheme of the patient can be understood, which helps the doctor to understand the treatment needs and goals of the patient and set the stimulation parameters according to the specific situation; according to the stimulation parameter set corresponding to each electrode lead, the stimulation result of each electrode lead can be calculated, and the effect of stimulation can be evaluated according to the stimulation result, which provides a basis for subsequent treatment adjustment; in the third display image, the stimulation result of each nucleus and each electrode lead can be visually displayed, helping the doctor to intuitively understand the distribution of the stimulation area and the stimulation intensity of each region point, which helps to evaluate the treatment effect and adjust the stimulation strategy. In summary, through the acquisition of the stimulation strategy, the calculation of the stimulation result and the visual display of the third display image, support is provided for individualized treatment, the doctor can evaluate the treatment effect according to the stimulation result, and adjust the stimulation strategy according to the visual display, thereby optimizing the effect of neural stimulation treatment and improving the curative effect and treatment satisfaction of the patient.

[0028] In some possible implementations, the at least one processor is configured to obtain the stimulation intensity of each region point in the following manner when executing the computer program:

[0029] Detect whether the region point is in the stimulation area of each electrode lead respectively, to obtain an electrode lead set in which the stimulation area includes the region point;

[0030] Input the position information of the region point and the pose information and the stimulation parameter set of all electrode leads in the electrode lead set into a flexible stimulation intensity model, to obtain the stimulation intensity corresponding to the region point.

[0031] The beneficial effects of the technical solution are that: by detecting whether each region point is in the stimulation area of each electrode lead, the electrode lead set containing the region point can be determined, which provides accurate input data for subsequent stimulation intensity calculation; by inputting the position information of the region point, the pose information of the electrode lead and the stimulation parameter set into the flexible stimulation intensity model, the stimulation intensity of each region point can be calculated; by calculating the stimulation intensity of each region point, a stimulation scheme can be customized for the patient; according to the stimulation intensity of the region point, the doctor can adjust the stimulation parameters to achieve more accurate and effective neural stimulation treatment, which helps to improve the targeting and curative effect of the treatment and better meet the treatment needs of the patient. In summary, through the stimulation area detection of the region point and the calculation of the flexible stimulation intensity model, a foundation is provided for individualized stimulation scheme, which can more accurately calculate the stimulation intensity of each region point, and helps to optimize neural stimulation treatment and provide support for doctor decision-making and patient treatment.

[0032] In some possible implementation manners, the training process of the flexible stimulation intensity model comprises the following steps:

[0033] obtaining a training set comprising a plurality of training data, each of the training data comprising sample position information and pose information of a sample electrode lead, a set of stimulation parameters, and labeled data of stimulation intensity corresponding to the sample position information and the pose information of the sample electrode lead and the set of stimulation parameters;

[0034] for each of the training data, performing the following processing:

[0035] inputting the sample position information and the pose information of the sample electrode lead and the set of stimulation parameters in the training data into a preset deep learning model to obtain predicted data of stimulation intensity corresponding to the sample position information and the pose information of the sample electrode lead and the set of stimulation parameters;

[0036] updating model parameters of the deep learning model according to the predicted data of stimulation intensity corresponding to the sample position information and the pose information of the sample electrode lead and the set of stimulation parameters and the labeled data;

[0037] detecting whether a preset training end condition is met; if yes, taking the trained deep learning model as the flexible stimulation intensity model; and if no, continuing to train the deep learning model using the next training data.

[0038] The technical scheme has the beneficial effects that by training the flexible stimulation intensity model, the relationship between the sample position information, the pose information of the electrode lead, the set of stimulation parameters and the stimulation intensity can be learned and modeled, the stimulation intensity can be accurately predicted, and the doctor and the technical personnel can better understand and control the stimulation treatment process; the flexible stimulation intensity model can be trained and optimized according to the condition and the requirement of each patient, the stimulation strategy can be realized by acquiring the set of stimulation parameters of each electrode lead and performing prediction, the accuracy and the effectiveness of the treatment can be ensured, the stimulation area and the distribution of the stimulation intensity can be better controlled by accurately predicting the stimulation intensity, and the accuracy and the precision of the treatment can be improved, and the performance and the accuracy of the model can be improved by continuously iterating the training process. In conclusion, the deep learning model can update the model parameters, the prediction ability of the model can be continuously optimized in the training process, accurate stimulation intensity prediction, individualized stimulation strategy and optimized treatment effect can be provided, and the technical scheme has important application value in the nucleus device technology.

[0039] In some possible implementation manners, the at least one processor is configured to obtain the stimulation intensity of each of the region points in the following manner when executing the computer program:

[0040] detect whether the stimulation regions of each two electrode leads intersect respectively;

[0041] When there are electrode leads with intersecting stimulation regions, for each two electrode leads with intersecting stimulation regions, the following processing is performed:

[0042] Obtain the intersecting region of the stimulation regions of the two electrode leads and the non-intersecting region corresponding to each of the two electrode leads respectively;

[0043] For each region point in the intersecting region, input the position information of the region point, the pose information and the stimulation parameter set of the two electrode leads into a double-stimulation intensity model to obtain the stimulation intensity of the region point;

[0044] For each region point in the non-intersecting region, input the position information of the region point, the pose information and the stimulation parameter set of the electrode lead corresponding to the region point into a single-stimulation intensity model to obtain the stimulation intensity of the region point.

[0045] The beneficial effects of the technical solution are that by detecting whether the stimulation regions of each two electrode leads intersect, it can be determined whether there are intersecting electrode leads, which helps to determine which electrode leads need to be further processed to obtain the stimulation intensity of the region point; for each region point in the intersecting region, by inputting the position information of the region point, the pose information and the stimulation parameter set of the two electrode leads into a double-stimulation intensity model, the stimulation intensity of the region point can be calculated; for each region point in the non-intersecting region, the position information of the region point, the pose information and the stimulation parameter set of the electrode lead corresponding to the region point can be input into a single-stimulation intensity model to calculate the stimulation intensity of the region point; by using the double-stimulation intensity model and the single-stimulation intensity model, the stimulation intensity of the region point can be calculated according to different situations, which can more accurately evaluate the stimulation intensity of the region point, thereby providing more refined adjustment and optimization for neural stimulation treatment. In summary, through stimulation region intersection detection and region point stimulation intensity calculation, the stimulation intensity of each region point can be more accurately obtained, which helps to realize individualized treatment optimization and improve the effect of neural stimulation treatment and the treatment experience of patients.

[0046] In a second aspect, the present application provides a nucleus labeling method, which comprises:

[0047] Obtaining medical image data of a patient's brain, and reconstructing a three-dimensional model of the patient's brain according to the medical image data;

[0048] Segmenting one or more nuclei from the medical image data to obtain a segmentation result of each of the nuclei;

[0049] According to the segmentation result of each nucleus and the three-dimensional model, a first display image is obtained, and the first display image is displayed using a wearable XR device, in which each nucleus is displayed separately, and the position and size of each nucleus are marked for a doctor wearing the XR device.

[0050] In a third aspect, the present application provides a wearable XR device, comprising:

[0051] The nucleus marking device according to any one of the above;

[0052] A camera is configured to capture real-time images and send the real-time images to the nucleus marking device.

[0053] A display screen is configured to provide a display function.

[0054] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by at least one processor to implement the steps of any one of the above methods or realize the functions of any one of the above electronic devices.

[0055] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by at least one processor to implement the steps of any one of the above methods or realize the functions of any one of the above electronic devices. BRIEF DESCRIPTION OF DRAWINGS

[0056] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0057] Figure 1 FIG. 1 is a flowchart of a nucleus marking method according to an embodiment of the present application.

[0058] Figure 2 FIG. 2 is a flowchart of obtaining a first display image according to an embodiment of the present application.

[0059] Figure 3 FIG. 3 is a flowchart of obtaining a stimulation strategy according to an embodiment of the present application.

[0060] Figure 4 FIG. 4 is a flowchart of obtaining a stimulation intensity according to an embodiment of the present application.

[0061] Figure 5 FIG. 5 is a structural block diagram of a wearable XR device according to an embodiment of the present application.

[0062] Figure 6 FIG. 6 is a structural block diagram of an electronic device according to an embodiment of the present application.

[0063] Figure 7 Fig. 1 is a structural schematic diagram of a computer program product according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the present application will be described below with reference to the accompanying drawings and specific embodiments of the present application. It should be noted that, under the premise of no conflict, the following described embodiments or technical features can be combined to form new embodiments.

[0065] In the embodiments of the present application, the words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any implementation or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other implementations or design schemes. Rather, the words such as "exemplary" or "for example" are used to present the relevant concept in a specific manner.

[0066] The first, second, and the like in the embodiments of the present application are only used for description and distinction of the description objects, and do not have order, and do not represent the special limitation of the quantity in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0067] The technical field and related terms of the embodiments of the present application will be briefly described below.

[0068] The implantable medical system includes an implantable neuro-electric stimulation system, an implantable cardiac electric stimulation system (also known as a cardiac pacemaker), an implantable drug delivery system (IDDS), and a lead adapter system, etc. The implantable neuro-electric stimulation system is, for example, a deep brain electric stimulation system (DBS), an implantable cortical nerve stimulation system (CNS), an implantable spinal cord electric stimulation system (SCS), an implantable sacral nerve electric stimulation system (SNS), an implantable vagus nerve electric stimulation system (VNS), etc.

[0069] An implantable neurostimulation system includes a stimulator (i.e., an implantable neurostimulator, a neurostimulation device) implanted in a patient and a programming device disposed outside the patient. That is, the stimulator is an implant, or the implant includes the stimulator. The relevant neuromodulation technology is mainly to implant an electrode (the electrode is in the form of an electrode lead, for example) at a specific site (i.e., a target site) of a tissue of a living body through stereotactic surgery, to deliver an electrical pulse to the target site through the electrode, to modulate the electrical activity and function of the corresponding neural structure and network, and thus to improve symptoms and relieve pain. The stimulator can include an IPG, an extension lead, and an electrode lead. The IPG (implantable pulse generator) is disposed in the patient's body and provides controllable electrical stimulation energy to the tissue in the body in response to the programming instructions sent by the programming device, relying on a sealed battery and a circuit. The IPG delivers one or more controllable specific electrical stimulations to a specific area of the tissue in the body through the extension lead and the electrode lead. The extension lead is used with the IPG as a transmission medium for the electrical stimulation signal, and delivers the electrical stimulation signal generated by the IPG to the electrode lead. The electrode lead delivers electrical stimulation to a specific area of the tissue in the body through a plurality of electrode contacts. The stimulator is provided with one or more electrode leads on one side or both sides, and the electrode lead is provided with a plurality of electrode contacts, which can be uniformly arranged or non-uniformly arranged on the circumference of the electrode lead. As an example, the electrode contacts can be arranged in a 4-row 3-column array (a total of 12 electrode contacts) on the circumference of the electrode lead. The electrode contacts can include stimulation electrode contacts and / or acquisition electrode contacts. The electrode contacts can have shapes such as sheet, ring, and dot, for example.

[0070] In some embodiments, the stimulated tissue in the body can be brain tissue of the patient, and the stimulated site can be a specific site of the brain tissue. When the type of disease of the patient is different, the stimulated site is generally different, and the number of stimulation contacts (single source or multiple sources) used, the use of one or more (single channel or multiple channels) specific electrical stimulation signals, and the stimulation parameter data are also different. The embodiments of the present application do not limit the applicable disease types, which can be applicable to deep brain stimulation (DBS), spinal cord stimulation (SCS), pelvic stimulation, stomach stimulation, peripheral nerve stimulation, and functional electrical stimulation. The disease types that can be treated or managed by DBS include but are not limited to: convulsive diseases (e.g., epilepsy), pain, migraine, mental diseases (e.g., major depressive disorder (MDD)), bipolar disorder, anxiety disorder, post-traumatic stress disorder, mild depression, obsessive-compulsive disorder (OCD), behavioral disorder, emotional disorder, memory disorder, mental state disorder, movement disorder (e.g., essential tremor or Parkinson's disease), Huntington's disease, Alzheimer's disease, drug addiction, autism, or other neurological or psychiatric diseases and injuries.

[0071] In the embodiments of the present application, when the program control device and the stimulator establish a program control connection, the program control device can adjust the stimulation parameters of the stimulator (or the stimulation parameters of the pulse generator, different stimulation parameters correspond to different electrical stimulation signals), or the stimulator can sense the electrophysiological activity of the patient to collect electrophysiological signals, and the stimulation parameters of the stimulator can be continuously adjusted based on the collected electrophysiological signals.

[0072] The stimulation parameters can include at least one of the following: electrode contact identification for delivering electrical stimulation (for example, 2# electrode contact and 3# electrode contact), frequency (for example, the number of electrical stimulation pulse signals per unit time 1s, unit Hz), pulse width (duration of each pulse, unit μs), amplitude (generally expressed in voltage, that is, the intensity of each pulse, unit V), timing (for example, continuous or burst, burst refers to a discontinuous timing behavior composed of multiple processes), stimulation mode (including one or more of current mode, voltage mode, timing stimulation mode and cycle stimulation mode), doctor control upper and lower limits (range adjustable by the doctor) and patient control upper and lower limits (range adjustable by the patient)

[0073] The stimulation intensity in the embodiments of the present application refers to the intensity of the electrical stimulation applied to the implantable neuro-electrical stimulation system. The measurement method of the electrical stimulation intensity can be current, voltage, charge amount, pulse width, pulse frequency, etc. The adjustment of the stimulation intensity can produce different effects and reactions on the nervous system, such as excitation or inhibition of neurons. When the current (in amperes) is used as the measurement method of the stimulation intensity, the current intensity can be 0.5mA, 1mA, 2mA, 4mA, 5mA, etc.

[0074] In one specific application scenario, the stimulation parameters of the stimulator can be adjusted in current mode or voltage mode.

[0075] The program control device can be a doctor program control device (i.e. a program control device used by a doctor) or a patient program control device (i.e. a program control device used by a patient). The doctor program control device can be, for example, a smart terminal device such as a tablet computer, a notebook computer, a desktop computer, a mobile phone, etc. loaded with program control software. The patient program control device can be, for example, a smart terminal device such as a tablet computer, a notebook computer, a desktop computer, a mobile phone, etc. loaded with program control software, and can also be other electronic devices with program control function (for example, a charger, a data acquisition device, etc. with program control function).

[0076] The embodiments of the present application do not limit the data interaction of the doctor programming device and the stimulator. When the doctor remotely programs, the doctor programming device can interact with the stimulator through the server and the patient programming device. When the doctor programs offline and face-to-face with the patient, the doctor programming device can interact with the stimulator through the patient programming device, and the doctor programming device can also directly interact with the stimulator.

[0077] In some embodiments, the patient programming device can include a host (communicating with the server) and a slave (communicating with the stimulator), which are communicatively connected. Wherein the doctor programming device can interact with the server through the 3G / 4G / 5G network, the server can interact with the host through the 3G / 4G / 5G network, the host can interact with the slave through the Bluetooth protocol / WIFI protocol / USB protocol, and the slave can interact with the stimulator through the 401MHz-406MHz operating frequency band / 2.4GHz-2.48GHz operating frequency band. The doctor programming device can directly interact with the stimulator through the 401MHz-406MHz operating frequency band / 2.4GHz-2.48GHz operating frequency band.

[0078] Artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation, etc.

[0079] Machine Learning (ML) is a multi-disciplinary field that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. A computer program can learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks T improves with experience E, measured by P. Machine learning is specifically concerned with the design and development of algorithms that allow computers to simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence.

[0080] Deep learning is a special kind of machine learning that uses nested levels of concepts to represent the world and achieve great functionality and flexibility, where each concept is defined as being associated with a simple concept, and more abstract representations are calculated in a less abstract way. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0081] XR device refers to an Extended Reality (XR) device, which is a device that integrates Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) technologies. XR technology aims to create a new digital experience that interacts with the real world. Virtual Reality (VR) completely immerses users in a virtual environment, making them feel as if they are there. Augmented Reality (AR) superimposes virtual content onto the real world, allowing users to interact with digital content in the real environment. Mixed Reality (MR) combines the features of virtual reality and augmented reality, allowing virtual content to interact and integrate with the real world in real time. XR devices usually include Head-Mounted Display (HMD), smart glasses, handheld devices, etc. These devices use sensors, cameras, positioning systems, etc. to capture real-time information about the user's head, gestures, position, etc. and present virtual or augmented content to the user in a realistic way.

[0082] Nucleus annotation is a task in neuroscience research aimed at annotating and identifying brain images or brain image data to determine and locate specific neural nuclei or brain regions. Nuclei are clusters of neurons with similar morphology, function or connection characteristics and are closely clustered in specific brain structures. Nucleus annotation is achieved by accurately outlining and marking the boundaries and regions of nuclei to provide detailed description and interpretation of brain structure and function. Nucleus annotation requires the combination of anatomical knowledge, brain atlas and brain model reference information to ensure the accuracy and consistency of annotation. The annotation process usually involves visual analysis of brain images, structure boundary identification and region segmentation, which can be completed by manual operation or assisted by automated tools.

[0083] With the development of technology and social progress, in the medical field, nucleus annotation aims to help doctors accurately locate and annotate nucleus structures in the brain. Nuclei are collections of specific regions in the brain that are crucial for cognitive, motor control, and emotional regulation functions. Accurate annotation of nuclei can help doctors better understand the brain structure of patients in surgical planning, neurological disease diagnosis and treatment. At the same time, with the development of virtual reality (VR) and augmented reality (AR) technology, wearable XR devices such as AR glasses have become a powerful tool for medical image visualization and navigation. By superimposing computer-generated images on real-world visual scenes, XR devices can provide an intuitive interface between doctors and patients' brains, making medical operations more precise and safe.

[0084] Currently, nucleus annotation and XR device application are relatively new technologies and are still in the development and improvement stage. In some areas or medical institutions, these technologies may not have been widely adopted or may be limited to specific research institutions or high-end medical centers. Based on this, the present application provides a nucleus annotation device, a wearable XR device, a nucleus annotation method, a computer readable storage medium and a computer program product to improve the prior art.

[0085] The scheme provided by the embodiments of the present application relates to the technical field of deep brain stimulation, XR devices, computer vision and deep learning, and in particular to a nucleus annotation device, a wearable XR device, a nucleus annotation method, a computer readable storage medium and a computer program product, which are specifically described through the following embodiments. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0086] (Nucleus annotation device)

[0087] The embodiment of the present application provides a nucleus annotation device, which comprises a memory and at least one processor, the memory stores a computer program, and the at least one processor is configured to implement the following steps when executing the computer program:

[0088] obtaining medical image data of a brain of a patient, and reconstructing a three-dimensional model of the brain of the patient according to the medical image data;

[0089] segmenting one or more nuclei from the medical image data to obtain a segmentation result of each of the nuclei;

[0090] obtaining a first display image according to the segmentation result of each of the nuclei and the three-dimensional model, and displaying the first display image using a wearable XR device, in which each of the nuclei is displayed separately, and the position and size of each of the nuclei are labeled for a doctor wearing the XR device.

[0091] In the embodiments of the present application, the medical device used to obtain the medical image data may be, for example, a CT device, an MR device, a PET device, an X-ray device, a PET-CT device, a PET-MR device, etc., and the medical image data may be, for example, CT data, MR data, PET data, X-ray data, PET-CT data, PET-MR data, etc. CT (Computed Tomography) is an electronic computed tomography, MR (Magnetic Resonance) is a magnetic resonance, and PET (Positron Emission Tomography) is a positron emission tomography.

[0092] Thus, by obtaining the medical image data of the brain of the patient and performing three-dimensional model reconstruction and nucleus segmentation, each nucleus can be accurately located and segmented, and a high-precision labeling result is provided for the doctor. By using the wearable XR device, the position and size of each nucleus can be displayed in real time in the first display image, providing an intuitive labeling experience for the doctor wearing the XR device. The doctor can directly observe and label the nuclei, improving the convenience and accuracy of the operation. In summary, the nucleus labeling device provides an accurate, real-time, and convenient nucleus labeling experience for the doctor by combining three-dimensional model reconstruction, nucleus segmentation, and the use of wearable XR devices, which helps to improve the efficiency and accuracy of medical work and promotes the progress of medical research and clinical practice.

[0093] For example, there is a patient A who is undergoing a brain imaging examination. Through computed tomography (CT) technology, the medical image data of A's brain is obtained, which is stored in the memory of the nucleus labeling device.

[0094] First, the nucleus labeling device reads the computer programs from the memory and starts at least one processor to execute the programs. The device processes the medical image data and reconstructs a three-dimensional model of A's brain according to the data;

[0095] Then, the nuclear marker device segments the medical image data to separate one or more nuclear medical image data from the brain image;

[0096] Finally, according to the segmentation result of each nuclear and the three-dimensional model, the nuclear marker device generates a display image, which is transmitted to the wearable XR device for the doctor to use. In this display image, each nuclear is displayed separately, and its position and size are labeled using specific colors or markers, such as the thalamus displayed in blue, the amygdala displayed in red, and the ventricle displayed in green.

[0097] By wearing the XR device, the doctor can observe the brain image of A in real-time view and intuitively see the position and size of each nuclear.

[0098] Through the above steps, the working principle of the nuclear marker device is demonstrated, which provides a convenient and accurate working method by obtaining medical image data, reconstructing a three-dimensional model, segmenting nuclear and generating a display image, providing more accurate structural information for doctors to help them make decisions and operations during surgery or treatment.

[0099] In some embodiments, the plurality of nuclears includes one or more of the following: nucleus accumbens, anterior limb of internal capsule, subthalamic nucleus, ventral intermediate nucleus, medial globus pallidus, ventral internal capsule, ventral striatum, and superior lateral branch of medial forebrain bundle.

[0100] Therefore, by using the nuclear marker device, multiple nuclears can be accurately labeled and located, which helps doctors to better understand the brain structure and information; by segmenting and reconstructing the three-dimensional model of the patient's brain, combined with the segmentation result of the nuclear, more comprehensive and accurate medical image processing can be provided, which helps doctors to better understand the patient's brain condition and provides more reliable basis, so that doctors can more accurately locate the target area. In summary, by providing accurate nuclear labeling and visualization tools, the accuracy and efficiency of medical image processing and neurosurgery are improved, which helps to improve the success rate of surgery and the treatment effect of patients.

[0101] For example, when a doctor uses the nuclear marker device, the patient's brain is scanned using a magnetic resonance imaging (MRI) device to obtain medical image data, and the patient's brain MRI scan can produce high-resolution three-dimensional images. Through the processor and computer program in the nuclear marker device, the medical image data is analyzed and processed to segment specific nuclears, and the regions of the thalamic nucleus and the anterior limb of the internal capsule are extracted.

[0102] Based on the segmentation result and the three-dimensional model of the patient's brain, the nucleus labeling device generates a display image. The doctor can wear augmented reality (AR) glasses to view this image, in the display of the AR glasses, the doctor can see the model of the patient's brain, and the nuclei are displayed in different colors or markers.

[0103] Through the above steps, the nucleus labeling device can accurately label the positions and sizes of the nuclei such as the nucleus accumbens, the anterior limb of the internal capsule, the subthalamic nucleus, the ventral intermediate nucleus of the thalamus, the medial globus pallidus, the ventral internal capsule, the ventral striatum, and the superior lateral branch of the medial forebrain bundle. Through the display of the AR glasses, the doctor can intuitively observe and analyze the patient's brain structure, which is of great significance for neurosurgical planning, brain disease diagnosis and treatment, etc.

[0104] In some embodiments, the XR device employs an AR device, and the at least one processor is configured to acquire the first display image in the following manner when executing the computer program:

[0105] The real-time image is obtained by a camera of the AR device, and the real-time image contains the patient's brain and one or more electrode leads implanted in the patient's brain;

[0106] The real-time image and the three-dimensional model are registered to obtain a registration matrix;

[0107] The first display image is obtained according to the segmentation result of each nucleus, the registration matrix, and the real-time image.

[0108] Therefore, by capturing the real-time image of the patient's brain and the implanted electrode lead through the camera of the AR device, and registering it with the previously reconstructed three-dimensional model, an accurate image of the patient's brain can be obtained in real time, and it can be aligned with the model, providing accurate basic data for subsequent processing and display; by combining the segmentation result and the registration matrix with the real-time image, the first display image can be generated, in which each nucleus can be clearly displayed, and its position and size are labeled, the doctor can intelligently navigate and locate the nucleus on the real-time image, better understand the patient's brain structure, and provide accurate guidance. In summary, by using the AR device as the XR device and combining real-time image acquisition, registration, and nucleus segmentation result, the doctor is provided with intelligent navigation and augmented reality auxiliary display functions, which helps to improve the accuracy, individualization, and efficiency of the surgery.

[0109] For example, when a doctor uses the nucleus labeling device, first, the medical imaging data of the patient's brain is obtained using a magnetic resonance imaging (MRI) device, and then a three-dimensional model of the patient's brain is reconstructed and obtained based on these data, and then the required nuclei are automatically segmented and extracted;

[0110] Then the camera of the AR device is used to capture a real-time image of the patient's brain, which includes the patient's brain and one or more electrode leads implanted in the patient's brain. The camera of the AR device can capture a real-time image of the patient's head, showing the location of the patient's brain and the distribution of the electrode leads.

[0111] The real-time image is registered with the pre-generated three-dimensional brain model to obtain a registration matrix. The purpose of registration is to align the real-time image with the three-dimensional model so that they are consistent in the same coordinate system. Using computer vision techniques, the feature points in the real-time image are matched with the corresponding points in the three-dimensional model to obtain the registration matrix.

[0112] Using the segmentation results of each nucleus, the registration matrix and the real-time image, a display image is generated. In this display image, the nuclei are labeled and displayed overlaid with the real-time image. The doctor can observe this image through the display screen of the AR device and obtain information about the location of the nuclei. For example, the nuclei can be displayed in different colors or marked areas, and the doctor can see the exact location and size of the nuclei in the patient's brain.

[0113] Through the above steps, the nucleus device uses an XR device (such as an AR device) to capture a real-time image using the camera of the AR device and registers it with a three-dimensional model to obtain a display image. The nucleus device is used to label the location and size of the nuclei to assist in the research and analysis of the patient's brain during diagnosis and treatment.

[0114] In some embodiments, the at least one processor, when configured to execute the computer program, also implements the following steps:

[0115] During the implantation of the electrode lead, according to the real-time image, real-time pose information of a target electrode lead currently implanted is obtained;

[0116] According to the real-time pose information, it is detected whether the target electrode lead deviates from its corresponding preset implantation path;

[0117] If it deviates, a second display image is obtained according to the segmentation result of the target target point corresponding to the target electrode lead, the registration matrix, the real-time image, the position information of the target electrode lead and the corresponding preset implantation path, and the AR device is used to display the second display image to assist the doctor in implanting the target electrode lead according to the preset implantation path. In the second display image, the target electrode lead, its corresponding target target point and preset implantation path are highlighted, and the target target point is one of the nuclei.

[0118] Thus, by using real-time images, the current position and attitude information of the electrode lead being implanted can be obtained, enabling the doctor to understand the position and direction of the lead in real time, thereby performing accurate operation and adjustment; by comparing the real-time pose information of the target electrode lead with the preset implantation path, it can be detected whether the lead deviates from the expected position, which helps the doctor to timely discover and correct the deviation or error that may occur in the lead implantation process; according to the segmentation result of the target target point corresponding to the target lead, the registration matrix, the real-time image and the lead position information, a second display image is generated, which can provide more intuitive navigation information to help the doctor implant the target electrode lead accurately according to the preset implantation path; by displaying the second display image on the XR device, the doctor can obtain visual feedback of augmented reality, and the highlight display of the lead, the target target point and the preset implantation path helps the doctor to accurately locate the target position in actual operation and ensures the correct implantation of the lead. In summary, the technical scheme provides real-time navigation and assistance, enabling the doctor to implant the target electrode lead more accurately, improving the implantation precision, reducing the operation risk and ensuring the achievement of the treatment effect.

[0119] For example, assume that a patient needs to undergo deep brain stimulation treatment and needs to implant an electrode lead into a patient's brain nucleus, one of which is a target nucleus;

[0120] First, use a medical imaging device to obtain medical imaging data of the patient's brain, such as CT scans and MRI, and use a real-time camera or other image acquisition device to obtain real-time images of the patient's brain during the implantation of the electrode lead;

[0121] Next, according to the real-time image, the real-time pose information of the target electrode lead being implanted is obtained, and the position and attitude of the target electrode lead are extracted through computer vision technology;

[0122] Then, according to the real-time pose information, it is detected whether the target electrode lead deviates from its preset implantation path, and if the target electrode lead deviates from the preset implantation path, a display image is obtained according to the segmentation result of the target target point corresponding to the target electrode lead, the registration matrix, the real-time image, the position information of the target electrode lead and the preset implantation path, etc. This display image will be displayed to the doctor through the AR device to help the doctor implant the target electrode lead according to the preset implantation path;

[0123] In this display image, the target electrode lead, its corresponding target target point and the preset implantation path can be highlighted. In this way, the doctor can observe the display image on the AR device to assist the process of implanting the target electrode lead. The AR device can superimpose the image on the actual brain image of the patient, enabling the doctor to intuitively see the position of the target electrode lead and the implantation path.

[0124] By the above steps, it can be clearly seen how to obtain a display image according to the real-time image, real-time pose information, target target point segmentation result, registration matrix, position information of the target electrode lead, and preset implantation path, and assist the doctor in implanting the target electrode lead using the AR device. This process can improve the accuracy and safety of the surgery, ensuring that the target electrode lead is correctly implanted in the target nucleus.

[0125] In some embodiments, the at least one processor, when configured to execute the computer program, also implements the following steps:

[0126] After implanting one or more electrode leads, obtain a stimulation strategy corresponding to the patient, the stimulation strategy including a set of stimulation parameters corresponding to each of the electrode leads;

[0127] According to the set of stimulation parameters corresponding to each of the electrode leads, obtain a stimulation result of each of the electrode leads, the stimulation result being used to indicate a stimulation area and a stimulation intensity of each area point in the stimulation area;

[0128] According to the segmentation result of each of the nuclei, the registration matrix, the real-time image, and the stimulation result of each of the electrode leads, obtain a third display image, and display the third display image using the AR device, in which each of the nuclei and the stimulation result of each of the electrode leads are visually displayed.

[0129] Thus, by obtaining a stimulation strategy corresponding to the patient, including a set of stimulation parameters corresponding to each of the electrode leads, the individualized stimulation plan of the patient can be understood, which helps the doctor to understand the treatment needs and goals of the patient and to set the stimulation parameters according to the specific circumstances; according to the set of stimulation parameters corresponding to each of the electrode leads, the stimulation result of each of the electrode leads can be calculated, the stimulation result being used to indicate a stimulation area and a stimulation intensity of each area point in the stimulation area, which can be used to evaluate the stimulation effect and provide a basis for subsequent treatment adjustment; according to the segmentation result of the nuclei, the registration matrix, the real-time image, and the stimulation result of the electrode leads, a third display image is obtained, in which the stimulation result of each of the nuclei and each of the electrode leads can be visually displayed, helping the doctor to intuitively understand the distribution of the stimulation area and the stimulation intensity of each area point, which helps to evaluate the treatment effect and adjust the stimulation strategy. In summary, through the acquisition of the stimulation strategy, the calculation of the stimulation result, and the visual display of the third display image, individualized treatment is supported, the doctor can evaluate the treatment effect according to the stimulation result, and adjust the stimulation strategy according to the visual display, thereby optimizing the effect of neurostimulation treatment and improving the efficacy and treatment satisfaction of the patient.

[0130] For example, assume that a patient is undergoing deep brain stimulation therapy using a nucleus device, and he has implanted multiple electrode leads into his brain nuclei;

[0131] First, the stimulation strategy corresponding to the patient needs to be obtained, which includes a set of stimulation parameters such as stimulation frequency, pulse width, and current intensity for each electrode lead, which will be used to control the stimulation process of the electrode lead;

[0132] Next, according to the set of stimulation parameters corresponding to each electrode lead, the stimulation result of each electrode lead is obtained. The stimulation result is used to indicate the stimulation area and the stimulation intensity of each region point in the stimulation area. For each electrode lead, the stimulation area can be calculated according to the set of stimulation parameters, and the stimulation intensity of each region point can be determined;

[0133] After obtaining the stimulation intensity, the display image can be obtained using the segmentation result of the nucleus, the registration matrix, the real-time image, and the stimulation result of each electrode lead. The display image will be used to visualize the stimulation result of the nucleus and the electrode lead, register the segmentation result of the nucleus with the real-time image, and then superimpose the stimulation result of each electrode lead onto the image. The display image is displayed using an AR device, and the doctor can intuitively observe the stimulation result of each nucleus and each electrode lead on the real-time image.

[0134] Through the above steps, it can be clearly seen how to obtain the stimulation intensity according to the patient's stimulation strategy, the set of stimulation parameters of the electrode lead, the segmentation result of the nucleus, the registration matrix, and the real-time image, and visualize it using an AR device. This process can help doctors better understand the stimulation effect to adjust the treatment plan and optimize the treatment effect.

[0135] In some embodiments, the at least one processor is configured to obtain the stimulation intensity of each region point in the following manner when executing the computer program:

[0136] Detect whether the region point is in the stimulation area of each electrode lead respectively to obtain a set of electrode leads whose stimulation areas include the region point;

[0137] Input the position information of the region point and the pose information, the set of stimulation parameters of all electrode leads in the set of electrode leads into a flexible stimulation intensity model to obtain the stimulation intensity corresponding to the region point.

[0138] In addition, when a region point is not in the stimulation area of any electrode lead, the stimulation intensity corresponding to the region point can be set to a default value, for example, 0.

[0139] Thus, by detecting whether each region point is within the stimulation region of each electrode lead, the set of electrode leads containing the region point can be determined, providing accurate input data for subsequent stimulation intensity calculation; by inputting the position information of the region point, the pose information of the electrode lead, and the set of stimulation parameters into the flexible stimulation intensity model, the stimulation intensity of each region point can be calculated; by calculating the stimulation intensity of each region point, a customized stimulation plan can be provided for the patient; according to the stimulation intensity of the region point, the doctor can adjust the stimulation parameters to achieve more precise and effective neurostimulation treatment, which helps to improve the targeting and efficacy of treatment and better meet the treatment needs of the patient; by obtaining the stimulation intensity of each region point, it can be visualized and displayed. In summary, through the detection of the stimulation region of the region point and the calculation of the flexible stimulation intensity model, a foundation is provided for individualized stimulation plan, which can more accurately calculate the stimulation intensity of each region point, help to optimize neurostimulation treatment, and provide more intuitive information through visual display, supporting doctor decision-making and patient treatment.

[0140] For example, there is a patient with epilepsy who needs deep brain stimulation treatment. During the treatment, the doctor implants two electrode leads into specific nuclei in the patient's brain, namely the left thalamic subnucleus and the right septal nucleus.

[0141] First, the medical imaging data of the patient's brain is obtained through medical imaging equipment, including CT scans and MRI. These data provide detailed information about the structure of the patient's brain.

[0142] Next, a three-dimensional model of the patient's brain is reconstructed using computer vision techniques, resulting in a three-dimensional model with two nuclei, one of which is the left thalamic subnucleus and the other is the right septal nucleus.

[0143] Then, the left thalamic subnucleus and the right septal nucleus are segmented from the medical imaging data, and these segmentation results are in the form of volume or surface representation of the nuclei.

[0144] During the implantation of the electrode leads, real-time images and positioning systems are used to obtain the real-time pose information of the target electrode lead currently implanted. The real-time image shows the two electrode leads in the patient's brain, which are connected to the left thalamic subnucleus and the right septal nucleus, respectively.

[0145] Next, it is detected whether each region point is within the stimulation region of each electrode lead. For example, select a region point located in the left thalamic subnucleus and a region point located in the right septal nucleus, and check whether these region points are within the stimulation region of the corresponding electrode lead.

[0146] Then, the position information of the region point, the electrode lead set containing the left thalamic subthalamic nucleus and right nucleus accumbens electrode lead wire pose information, and the stimulation parameter set are input into the flexible stimulation intensity model, which calculates the stimulation intensity of the region point according to the position, pose and stimulation parameter of the lead wire.

[0147] Finally, the stimulation intensity value corresponding to each region point, i.e., the left thalamic subthalamic nucleus region point and the right nucleus accumbens region point, is obtained.

[0148] Through the above steps, it can be clearly seen how to input the flexible stimulation intensity model according to the position of the region point, the pose information of the electrode lead set and the stimulation parameter to obtain the stimulation intensity of each region point. This process can help doctors accurately control deep brain stimulation therapy to improve the symptoms of patients.

[0149] In some embodiments, the training process of the flexible stimulation intensity model comprises:

[0150] Obtain a training set, the training set comprising a plurality of training data, each training data comprising a sample position information and a sample electrode lead pose information, a stimulation parameter set and a labeled data of stimulation intensity corresponding to the sample position information and the sample electrode lead pose information, the stimulation parameter set;

[0151] For each training data, the following processing is performed:

[0152] Input the sample position information and the sample electrode lead pose information, the stimulation parameter set in the training data into a preset deep learning model to obtain the prediction data of the stimulation intensity corresponding to the sample position information and the sample electrode lead pose information, the stimulation parameter set;

[0153] According to the prediction data and the labeled data of the stimulation intensity corresponding to the sample position information and the sample electrode lead pose information, the stimulation parameter set, the model parameters of the deep learning model are updated;

[0154] Detect whether a preset training end condition is met; if yes, the trained deep learning model is used as the flexible stimulation intensity model; if no, continue to use the next training data to train the deep learning model.

[0155] In some embodiments, historical data can be mined to obtain sample data in the training set. That is, these sample data can be obtained by collecting real-time images multiple times in a real environment. In addition, the sample data can also be automatically generated by using the generation network of the GAN model.

[0156] The GAN model is a generative adversarial network, which is composed of a generation network and a discrimination network. The generation network randomly samples from a latent space as input, and the output result needs to imitate the real samples in the training set as much as possible. The input of the discrimination network is the real sample or the output of the generation network, and the purpose is to distinguish the output of the generation network from the real sample as much as possible. The generation network tries to deceive the discrimination network. The two networks are in mutual opposition and constantly adjust the parameters, and the final purpose is to make the discrimination network unable to judge whether the output result of the generation network is real. Using the GAN model can generate a large amount of sample data for the training process of the flexible stimulation intensity model, which can effectively reduce the amount of original data collection and greatly reduce the cost of data collection and labeling.

[0157] The training process of the flexible stimulation intensity model is not limited in the embodiments of the present application, which may, for example, adopt a supervised learning training mode, or may adopt a semi-supervised learning training mode, or may adopt an unsupervised learning training mode.

[0158] When the supervised learning or semi-supervised learning training mode is adopted, the acquisition mode of the labeled data is not limited in the embodiments of the present application, which may, for example, adopt a manual labeling mode, or may adopt an automatic labeling or semi-automatic labeling mode. When the sample data is collected in a real environment, the real data can be obtained from historical data by keyword extraction as labeled data.

[0159] The training end condition in the training process of the flexible stimulation intensity model is not limited in the embodiments of the present application, which may, for example, be that the number of training reaches a preset number (the preset number may be 1, 3, 10, 100, 1000, 10000, etc.), or may be that the training data in the training set are trained once or more, or may be that the total loss value obtained by this training is not greater than a preset loss value.

[0160] Thus, by training the flexible stimulation intensity model, the relationship between sample position information, electrode lead pose information, and stimulation parameter set and stimulation intensity can be learned and modeled, enabling accurate prediction of stimulation intensity and helping doctors and technicians better understand and control the stimulation treatment process; the flexible stimulation intensity model can be trained and optimized according to the condition and needs of each patient, and by obtaining the stimulation parameter set of each electrode lead and making predictions, personalized stimulation strategies can be achieved to ensure the accuracy and effectiveness of treatment; by accurately predicting the stimulation intensity, the distribution of the stimulation area and the stimulation intensity can be better controlled, thereby improving the accuracy and precision of treatment and maximizing the effectiveness of treatment; through continuous iterative training process, the performance and accuracy of the model can be improved. In summary, using a deep learning model can update the model parameters to continuously optimize the prediction ability of the model, providing accurate stimulation intensity prediction, personalized stimulation strategies, and optimized treatment effects, which has important application value in the nucleus device technology.

[0161] In some embodiments, the at least one processor is configured to obtain the stimulation intensity of each of the region points in the following manner when executing the computer program:

[0162] Detect whether the stimulation areas of each two electrode leads intersect, respectively;

[0163] When there are electrode leads with intersecting stimulation areas, for each two electrode leads with intersecting stimulation areas, the following processing is performed:

[0164] Obtain the intersection area of the stimulation areas of the two electrode leads and the non-intersecting area corresponding to each of the two electrode leads, respectively;

[0165] For each region point in the intersection area, input the position information of the region point and the pose information and stimulation parameter set of the two electrode leads into the double-stimulation intensity model to obtain the stimulation intensity of the region point;

[0166] For each region point in the non-intersecting area, input the position information of the region point and the pose information and stimulation parameter set of the electrode lead corresponding to the region point into the single-stimulation intensity model to obtain the stimulation intensity of the region point.

[0167] The training processes of the double-stimulation intensity model and the single-stimulation intensity model are similar to that of the flexible stimulation intensity model, which will not be described here. It should be noted that these models are all used to predict stimulation intensity and can be trained based on an initial model using deep learning or machine learning. Different names are used to distinguish different stimulation intensity models. The "flexible" in the flexible stimulation intensity model refers to the number of electrode leads in the input data of the model, which can be 1, 2, or more. Therefore, for any region point in the stimulation area of the electrode lead, regardless of the number of electrode leads in the stimulation area, the flexible stimulation intensity model can predict the stimulation intensity of the region point corresponding to the input data with the relevant information of the electrode lead. The double-stimulation intensity model and the single-stimulation intensity model divide the region points in the stimulation area of the electrode lead into two types: the intersection region of the stimulation area of two electrode leads (i.e., the intersection region of the stimulation area of two electrode leads), or the stimulation area of a single electrode lead (i.e., the non-intersection region corresponding to each of the two electrode leads). For these two cases, the double-stimulation intensity model and the single-stimulation intensity model are used to predict the stimulation intensity of the region point, which has a simpler model structure, fewer model parameters, and higher data processing efficiency.

[0168] Therefore, by detecting whether the stimulation areas of each two electrode leads intersect, it can be determined whether there are intersecting electrode leads. This helps to determine which electrode leads need to be further processed to obtain the stimulation intensity of the region point. For each region point in the intersection region, the position information of the region point, the pose information of the two electrode leads, and the stimulation parameter set are input into the double-stimulation intensity model to calculate the stimulation intensity of the region point. For each region point in the non-intersection region, the position information of the region point, the pose information of the electrode lead corresponding to the region point, and the stimulation parameter set are input into the single-stimulation intensity model to calculate the stimulation intensity of the region point. By using the double-stimulation intensity model and the single-stimulation intensity model, the stimulation intensity of the region point can be calculated according to different situations. This can more accurately evaluate the stimulation intensity of the region point, thereby providing more refined adjustment and optimization for neural stimulation treatment. By calculating the stimulation intensity of each region point, individualized treatment can be supported. In summary, through stimulation area intersection detection and region point stimulation intensity calculation, the technical solution can more accurately obtain the stimulation intensity of each region point. This helps to achieve individualized treatment optimization and improve the effectiveness of neural stimulation treatment and the treatment experience of patients.

[0169] For example, there is a current nucleus device for deep brain stimulation treatment, which includes two electrode leads: electrode lead A and electrode lead B. By analyzing the shape and stimulation parameters of the electrode leads, it is determined whether the stimulation areas of electrode lead A and electrode lead B intersect.

[0170] Assuming that the stimulation areas of electrode lead A and electrode lead B intersect, the intersection area, i.e. the part where the stimulation areas of electrode lead A and electrode lead B overlap, is obtained, and their respective non-intersection areas are obtained.

[0171] At a region point P in the intersection area, record its position information, and input the position information of the region point P, the pose information and stimulation parameter set of electrode lead A and electrode lead B into the double-stimulation intensity model. The double-stimulation intensity model will consider the stimulation parameters and positional relationship of electrode lead A and electrode lead B, and calculate the stimulation intensity of region point P.

[0172] At a region point Q in the non-intersection area, record its position information, and input the position information of the region point Q, the pose information and stimulation parameter set of electrode lead A into the single-stimulation intensity model. The single-stimulation intensity model only considers the stimulation parameters and position information of electrode lead A, and calculates the stimulation intensity of region point Q.

[0173] Through the above steps, the double-stimulation intensity value of region point P in the intersection area and the single-stimulation intensity value of region point Q in the non-intersection area can be obtained. These stimulation intensity values can be used to evaluate the treatment effect, optimize the stimulation parameters, and assist the doctor in precise positioning and adjustment of deep brain stimulation treatment.

[0174] In one specific application scenario, the embodiment of the present application also provides a nucleus labeling device, which comprises a memory and at least one processor, the memory stores a computer program, and the at least one processor is configured to implement the following steps when executing the computer program:

[0175] Obtain medical image data of the patient's brain, and reconstruct a three-dimensional model of the patient's brain according to the medical image data;

[0176] Segment one or more nuclei from the medical image data, and the plurality of nuclei includes one or more of the following: nucleus accumbens, anterior limb of internal capsule, subthalamic nucleus, ventral intermediate nucleus of thalamus, medial globus pallidus, ventral internal capsule, ventral striatum and upper lateral branch of medial forebrain bundle, to obtain a segmentation result of each nucleus;

[0177] Obtain a real-time image through a camera of the AR device, and the real-time image contains the patient's brain and one or more electrode leads implanted in the patient's brain;

[0178] register the real-time image and the three-dimensional model to obtain a registration matrix;

[0179] According to the segmentation result of each nucleus, the registration matrix and the real-time image, a first display image is obtained, and the first display image is displayed using a wearable XR device, in which each nucleus is displayed separately, and the position and size of each nucleus are marked for a doctor wearing the XR device.

[0180] During the implantation of the electrode lead, real-time pose information of a target electrode lead currently implanted is obtained according to the real-time image;

[0181] According to the real-time pose information, it is detected whether the target electrode lead deviates from a preset implantation path corresponding to the target electrode lead;

[0182] If deviated, a second display image is obtained according to the segmentation result of a target target point corresponding to the target electrode lead, the registration matrix, the real-time image, the position information of the target electrode lead and the corresponding preset implantation path, and the second display image is displayed using the AR device to assist the doctor to implant the target electrode lead according to the preset implantation path, in which the target electrode lead, the corresponding target target point and the preset implantation path are highlighted, and the target target point is one of the nuclei.

[0183] After implanting one or more electrode leads, a stimulation strategy corresponding to the patient is obtained, and the stimulation strategy includes a set of stimulation parameters corresponding to each electrode lead; according to the set of stimulation parameters corresponding to each electrode lead, a stimulation result of each electrode lead is obtained, which is used to indicate a stimulation area and a stimulation intensity of each region point in the stimulation area;

[0184] According to the segmentation result of each nucleus, the registration matrix, the real-time image and the stimulation result of each electrode lead, a third display image is obtained, and the third display image is displayed using the AR device, in which the stimulation result of each nucleus and each electrode lead is visually displayed;

[0185] It is respectively detected whether the region point is in the stimulation area of each electrode lead to obtain an electrode lead set in which the stimulation area includes the region point;

[0186] The position information of the region point and the pose information and the set of stimulation parameters of all electrode leads in the electrode lead set are input into a flexible stimulation intensity model to obtain a stimulation intensity corresponding to the region point;

[0187] detect whether the stimulation regions of each two electrode leads intersect respectively;

[0188] When there are electrode leads with intersecting stimulation regions, for each two electrode leads with intersecting stimulation regions, the following processing is performed:

[0189] obtain the intersecting region of the stimulation regions of the two electrode leads and the non-intersecting region corresponding to each of the two electrode leads respectively;

[0190] for each region point in the intersecting region, input the position information of the region point, the pose information and the stimulation parameter set of the two electrode leads into the dual-stimulation intensity model to obtain the stimulation intensity of the region point;

[0191] for each region point in the non-intersecting region, input the position information of the region point, the pose information and the stimulation parameter set of the electrode lead corresponding to the region point into the single-stimulation intensity model to obtain the stimulation intensity of the region point.

[0192] The training process of the flexible stimulation intensity model comprises:

[0193] obtain a training set, the training set comprising a plurality of training data, each of the training data comprising a sample position information, a pose information and a stimulation parameter set of a sample electrode lead, and labeled data of a stimulation intensity corresponding to the sample position information, the pose information and the stimulation parameter set of the sample electrode lead;

[0194] for each of the training data, the following processing is performed:

[0195] input the sample position information and the pose information and the stimulation parameter set of the sample electrode lead in the training data into a preset deep learning model to obtain predicted data of the stimulation intensity corresponding to the sample position information and the pose information and the stimulation parameter set of the sample electrode lead;

[0196] update the model parameters of the deep learning model according to the predicted data and the labeled data of the stimulation intensity corresponding to the sample position information and the pose information and the stimulation parameter set of the sample electrode lead;

[0197] detect whether a preset training end condition is met; if yes, the trained deep learning model is taken as the flexible stimulation intensity model; if no, continue to train the deep learning model using the next training data.

[0198] For example, assume that there is a nucleus labeling device designed to assist doctors in positioning and implanting electrode leads in deep brain stimulation therapy.

[0199] First, the doctor obtains image data of the patient's brain using magnetic resonance imaging (MRI) or other medical imaging techniques, including its anatomical structure and the distribution of nuclei, which are used for subsequent processing and analysis;

[0200] Next, a three-dimensional model of the patient's brain is reconstructed by processing and analyzing the medical image data, which is used for subsequent registration and positioning to ensure accurate matching;

[0201] Then, each nucleus, such as the nucleus accumbens, the anterior limb of the internal capsule, and the subthalamic nucleus, is automatically segmented from the medical image data. The segmentation results of these nuclei will be used in the subsequent positioning and labeling process;

[0202] During the operation, the camera of the augmented reality (AR) device captures real-time images of the patient's brain and the implanted electrode leads. These real-time images are registered with the previously obtained three-dimensional model to generate a registration matrix, ensuring accurate correspondence between the real-time images and the model;

[0203] Based on the segmentation results of the nuclei, the registration matrix, and the real-time images, a display image is generated, which is presented to the doctor through the wearable AR device. In the display image, each nucleus is displayed separately, and the doctor can wear the AR device on their head to visually observe the location and size of each labeled nucleus;

[0204] During the electrode lead implantation process, the real-time pose information of the target electrode lead being implanted is obtained from the real-time images. By comparing the real-time pose information with the pre-set implantation path, it is detected whether the target electrode lead deviates from the pre-set path. If there is a deviation, a display image is generated again, in which the target electrode lead, the target target point, and the pre-set implantation path are highlighted to assist the doctor in implanting the target electrode lead according to the pre-set path;

[0205] After implanting one or more electrode leads, the corresponding stimulation strategy of the patient is obtained, including the stimulation parameter set corresponding to each electrode lead, such as stimulation frequency, intensity, etc.

[0206] According to the stimulation parameter set of each electrode lead, the stimulation result of each electrode lead is calculated, indicating the stimulation area and the stimulation intensity of the region point. Combined with the segmentation results of the nuclei, the registration matrix, and the real-time images, an updated display image is generated and displayed on the AR device, which visualizes each nucleus and the stimulation result of each electrode lead;

[0207] For each region point, it is detected whether it is located within the stimulation area of the electrode lead to determine the set of electrode leads whose stimulation area includes the region point. The position information of the region point and the pose information and stimulation parameter set of the involved electrode leads are input into the flexible stimulation intensity model to calculate the stimulation intensity of the region point.

[0208] Alternatively, whether the stimulation areas of each two electrode leads intersect is detected, if there are intersecting electrode leads, the double stimulation intensity and the single stimulation intensity are calculated respectively for each region point in the intersecting region and the non-intersecting region, the region points in the intersecting region are calculated stimulation intensity through the double stimulation intensity model, and the region points in the non-intersecting region are calculated stimulation intensity through the single stimulation intensity model.

[0209] Through the above steps, the nucleus labeling device can help the doctor to realize the positioning and labeling of the nucleus, assist in implanting the target electrode lead, and provide the stimulation intensity calculation of the stimulation area and the region point, thereby improving the accuracy and personalized adjustment of deep brain stimulation treatment.

[0210] (Nucleus labeling method)

[0211] The embodiment of the application further provides a nucleus labeling method, and specific embodiments and achieved technical effects of the method are consistent with those of the above-mentioned device embodiments, and some contents will not be described herein again.

[0212] Referring to Figure 1 , Figure 1 is a flowchart of a nucleus labeling method provided by an embodiment of the application.

[0213] The application provides a nucleus labeling method, which comprises:

[0214] Step S101: acquiring medical image data of a brain of a patient, and reconstructing a three-dimensional model of the brain of the patient according to the medical image data;

[0215] Step S102: segmenting one or more nuclei from the medical image data to obtain a segmentation result of each of the nuclei;

[0216] Step S103: acquiring a first display image according to the segmentation result of each of the nuclei and the three-dimensional model, and displaying the first display image by using a wearable XR device, wherein each of the nuclei is displayed separately in the first display image, and the position and size of each of the nuclei are labeled for a doctor wearing the XR device.

[0217] In some embodiments, the plurality of nuclei comprises one or more of the following: the nucleus accumbens, the anterior limb of the internal capsule, the subthalamic nucleus, the ventral intermediate nucleus of the thalamus, the medial globus pallidus, the ventral internal capsule, the ventral striatum, and the superior lateral branch of the medial forebrain bundle.

[0218] Referring to Figure 2 , Figure 2 is a flowchart of acquiring a first display image provided by an embodiment of the application.

[0219] In some embodiments, the XR device is an AR device, and the process of obtaining the first display image comprises:

[0220] Step S201: obtaining a real-time image by a camera of the AR device, the real-time image containing a brain of the patient and one or more electrode leads implanted in the brain of the patient;

[0221] Step S202: registering the real-time image and the three-dimensional model to obtain a registration matrix;

[0222] According to the segmentation result of each of the nuclei, the registration matrix and the real-time image, the first display image is obtained.

[0223] In some embodiments, the method further comprises:

[0224] In the process of implanting the electrode lead, real-time pose information of a target electrode lead currently implanted is obtained according to the real-time image;

[0225] According to the real-time pose information, it is detected whether the target electrode lead deviates from a preset implantation path corresponding to the target electrode lead;

[0226] If deviated, a second display image is obtained according to the segmentation result of a target target point corresponding to the target electrode lead, the registration matrix, the real-time image, position information of the target electrode lead and the corresponding preset implantation path, and the AR device is used to display the second display image to assist the doctor to implant the target electrode lead according to the preset implantation path, in the second display image, the target electrode lead, the target target point corresponding to the target electrode lead and the preset implantation path are highlighted, and the target target point is one of the nuclei.

[0227] Referring to Figure 3 , Figure 3 is a flowchart of a process of obtaining a stimulation strategy provided by an embodiment of the present application.

[0228] In some embodiments, the method further comprises:

[0229] Step S301: after implanting one or more electrode leads, obtaining a stimulation strategy corresponding to the patient, the stimulation strategy comprising a set of stimulation parameters corresponding to each of the electrode leads;

[0230] Step S302: according to the set of stimulation parameters corresponding to each of the electrode leads, obtaining a stimulation result of each of the electrode leads, the stimulation result being used to indicate a stimulation area and a stimulation intensity of each area point in the stimulation area;

[0231] Step S303: According to the segmentation result of each nucleus, the registration matrix, the real-time image and the stimulation result of each electrode lead, a third display image is obtained, and the third display image is displayed using the AR device, in which the stimulation result of each nucleus and each electrode lead is visually displayed.

[0232] Referring to Figure 4 , Figure 4 is a flowchart of a process for obtaining a stimulation intensity provided by an embodiment of the present application.

[0233] In some embodiments, the process of obtaining the stimulation intensity of each region point includes:

[0234] Step S401: Detect whether each region point is in the stimulation region of each electrode lead respectively, to obtain an electrode lead set whose stimulation region includes the region point;

[0235] Step S402: Input the position information of the region point and the pose information and stimulation parameter set of all electrode leads in the electrode lead set into a flexible stimulation intensity model, to obtain the stimulation intensity corresponding to the region point.

[0236] In some embodiments, the process of obtaining the stimulation intensity of each region point includes:

[0237] Detect whether the stimulation regions of each two electrode leads intersect respectively;

[0238] When there are electrode leads whose stimulation regions intersect, for each two electrode leads whose stimulation regions intersect, the following processing is performed:

[0239] Obtain the intersection region of the stimulation regions of the two electrode leads and the non-intersection region corresponding to each of the two electrode leads respectively;

[0240] For each region point in the intersection region, input the position information of the region point and the pose information and stimulation parameter set of the two electrode leads into a double-stimulation intensity model, to obtain the stimulation intensity of the region point;

[0241] For each region point in the non-intersection region, input the position information of the region point and the pose information and stimulation parameter set of the electrode lead corresponding to the region point itself into a single-stimulation intensity model, to obtain the stimulation intensity of the region point.

[0242] (wearable XR device)

[0243] Referring to Figure 5 , Figure 5 is a structural block diagram of a wearable XR device provided by an embodiment of the present application.

[0244] The application provides a wearable XR device, which comprises:

[0245] The nuclear cluster labeling device according to any one of the above;

[0246] A camera is configured to acquire real-time images and send the real-time images to the nuclear cluster labeling device.

[0247] A display screen is configured to provide a display function.

[0248] In the embodiments of the application, the XR device can have the shape of glasses, a helmet or a hat, for example.

[0249] In addition, the XR device can further comprise one or more electrode pads, each of which is configured to contact the body surface of the doctor to detect the body state of the doctor. Specifically, a set of physiological parameters of the doctor is detected by the electrode pads, and the set of physiological parameters comprises one or more physiological parameters in electroencephalogram, electrocardiogram, electromyogram and electrooculogram; when one or more physiological parameters in the set of physiological parameters are not in a preset numerical range corresponding to the one or more physiological parameters, the doctor is prompted to be suspected to be in an unhealthy state, and the management personnel or other personnel are prompted to check the doctor to avoid sudden illness or sudden death of the doctor. The advantage of this is that for the doctor group with high professional pressure, there is a possibility of being on the operating table for a long time, and when the doctor continuously performs multiple operations or the time for a single operation is too long, the doctor's health is easily damaged or abnormal. By monitoring the health state of the doctor in real time by using the XR device, the health and safety of the doctor and the patient can be ensured.

[0250] Referring to Figure 6 , Figure 6 is a structural block diagram of an electronic device 10 provided by the embodiments of the application.

[0251] The electronic device 10 can comprise at least one memory 11, at least one processor 12 and a bus 13 connecting different platform systems.

[0252] The memory 11 can comprise a (computer) readable medium in the form of a volatile memory, such as a random access memory (RAM) 111 and / or a cache memory 112, and can further comprise a read-only memory (ROM) 113.

[0253] The memory 11 further stores a computer program, which can be executed by the processor 12 to enable the processor 12 to implement the steps of any one of the above methods.

[0254] The memory 11 can also include a utilities 114 having at least one program module 115, such as an operating system, one or more application programs, other program modules, and program data, and each or certain combination thereof can include implementation of a network environment.

[0255] Correspondingly, the processor 12 can execute the above computer program, and can execute the utilities 114.

[0256] The processor 12 can employ one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0257] The bus 13 can be one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus structures, and a proprietary bus to be used in conjunction with the various bus structures.

[0258] The electronic device 10 can also communicate with one or more external devices such as a keyboard or pointing device, a Bluetooth device, etc.; other devices that can or can not be handheld and that are peripheral as well as devices that are not in communication with the electronic device 10, and / or devices that enable communications between the electronic device 10 and other computing devices. Such communication can occur via the input / output interface 14. Also, the electronic device 10 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via a network adapter 15. The network adapter 15 can be any of a plurality of different types of adapters to enable the electronic device 10 to communicate with a network and / or another device or service. It should be appreciated that, although not shown, in some embodiments, the electronic device 10 can include additional hardware and / or software modules that can be used in connection with the electronic device 10, such as, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0259] (computer-readable storage medium)

[0260] The embodiments of the present application further provide a computer-readable storage medium, and specific embodiments thereof are consistent with the embodiments and the achieved technical effects of the above-mentioned method embodiments, and part of the content will not be described herein.

[0261] The computer readable storage medium stores a computer program, and the computer program is executed by at least one processor to implement the steps of any of the methods or implement the functions of any of the electronic devices.

[0262] The computer readable medium can be a computer readable signal medium or a computer readable storage medium. In embodiments of the present application, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus or device. The computer readable storage medium can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0263] The computer readable storage medium can include a data signal embodied in a carrier wave, which carries the readable program code. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable storage medium can also be any computer readable medium other than a transmission or propagation medium. The computer readable storage medium can be a recording medium, a computer readable storage medium, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The program code contained on the computer readable storage medium can be transmitted or propagated using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination of the above. The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, or the like, and conventional procedural programming languages such as the C programming language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0264] (computer program product)

[0265] The embodiment of the present application further provides a computer program product, and specific embodiments of the computer program product are consistent with the embodiments and the achieved technical effects of the above method embodiments, and part of the content will not be described herein again.

[0266] The computer program product comprises a computer program, and the computer program is executed by at least one processor to implement the steps of any one of the above methods or to implement the functions of any one of the above electronic devices.

[0267] Referring to Figure 7 , Figure 7 FIG. 1 is a structural schematic diagram of a computer program product provided by the embodiment of the present application.

[0268] The computer program product is used to implement the steps of any one of the above methods or to implement the functions of any one of the above electronic devices. The computer program product can adopt a portable compact disc read-only memory (CD-ROM) and comprises a program code, and can run on a terminal device such as a personal computer. However, the computer program product of the present application is not limited to this, and the computer program product can adopt any combination of one or more computer readable media.

[0269] The present application is described from the use purpose, the efficiency, the progress and the novelty, and meets the function improvement and the use requirements emphasized by the patent law. The above description and the attached drawings are only the preferred embodiments of the present application, and are not limited to the present application. Therefore, all the similar, identical, equivalent replacement or modification of the structure, device and features of the present application, and the patent application range of the present application, should be within the scope of the patent application protection.

Claims

1. A nucleus marking device, characterized in that: The nuclear group labeling apparatus includes a memory and at least one processor, wherein the memory stores a computer program, and the at least one processor is configured to implement the following steps when executing the computer program: Acquiring medical imaging data of a patient's brain, and reconstructing a three-dimensional model of the patient's brain based on the medical imaging data; Segmenting one or more nuclei from the medical image data to obtain a segmentation result of each nucleus; Acquire a first display image based on the segmentation result of each nucleus and the three-dimensional model, and display the first display image using a wearable XR device, wherein each nucleus is displayed separately in the first display image, and the position and size of each nucleus is marked for a doctor wearing the XR device; The acquiring of the first display image comprises: Capturing a real-time image through a camera of a wearable XR device, wherein the real-time image includes the patient's brain and one or more electrode leads implanted in the patient's brain; Registering the real-time image and the three-dimensional model to obtain a registration matrix; A first display image is acquired according to the segmentation result of each nuclear group, the registration matrix and the real-time image.

2. The nucleus marking device according to claim 1, characterized in that: The plurality of nuclei include one or more of the following: nucleus accumbens, anterior limb of internal capsule, subthalamic nucleus, ventral intermediate nucleus of the thalamus, internal part of the globus pallidus, ventral internal capsule, ventral striatum, and superior lateral ramus of the medial forebrain bundle.

3. The nucleus marking device according to claim 1, characterized in that: The XR device is an AR device, and the at least one processor is configured to acquire the first display image in the following manner when executing the computer program: Acquiring a real-time image through a camera of the AR device, wherein the real-time image includes the patient's brain and one or more electrode leads implanted in the patient's brain; Registering the real-time image and the three-dimensional model to obtain a registration matrix; The first display image is acquired according to the segmentation result of each nuclear group, the registration matrix and the real-time image.

4. The nucleus marking device according to claim 3, characterized in that: The at least one processor is configured to further implement the following steps when executing the computer program: During the process of implanting the electrode lead, obtaining real-time position information of the currently implanted target electrode lead according to the real-time image; Detecting, based on the real-time posture information, whether the target electrode wire deviates from its corresponding preset implantation path; If there is a deviation, a second display image is obtained based on the segmentation result of the target target corresponding to the target electrode wire, the alignment matrix, the real-time image, the position information of the target electrode wire and the corresponding preset implantation path, and the second display image is displayed using the AR device to assist the doctor in implanting the target electrode wire according to the preset implantation path. In the second display image, the target electrode wire and its corresponding target target and preset implantation path are highlighted, and the target target is one of the nuclear groups.

5. The nucleus marking device according to claim 3, characterized in that: The at least one processor is configured to further implement the following steps when executing the computer program: After implanting one or more electrode leads, obtaining a stimulation strategy corresponding to the patient, the stimulation strategy including a set of stimulation parameters corresponding to each electrode lead; Acquire a stimulation result for each electrode lead according to a stimulation parameter set corresponding to each electrode lead, wherein the stimulation result is used to indicate a stimulation area and a stimulation intensity of each area point in the stimulation area; A third display image is acquired based on the segmentation result of each nuclear group, the alignment matrix, the real-time image and the stimulation result of each electrode lead, and the third display image is displayed using the AR device. In the third display image, the stimulation result of each nuclear group and each electrode lead is visually displayed.

6. The nucleus marking device according to claim 5, characterized in that: The at least one processor is configured to obtain the stimulation intensity of each of the regional points in the following manner when executing the computer program: respectively detecting whether the regional point is within the stimulation area of ​​each electrode lead, so as to obtain an electrode lead set whose stimulation area includes the regional point; The position information of the regional point, the posture information of all electrode wires in the electrode wire set, and the stimulation parameter set are input into the flexible stimulation intensity model to obtain the stimulation intensity corresponding to the regional point.

7. The nucleus marking device according to claim 5, characterized in that: The at least one processor is configured to obtain the stimulation intensity of each of the regional points in the following manner when executing the computer program: Detect whether the stimulation areas of each two electrode wires intersect; When there are electrode leads with intersecting stimulation areas, the following process is performed for every two electrode leads with intersecting stimulation areas: Obtaining the intersection area of ​​the stimulation areas of the two electrode wires and the non-intersection areas corresponding to the two electrode wires; For each area point in the intersection area, inputting the position information of the area point, the posture information of the two electrode wires, and the stimulation parameter set into a dual stimulation intensity model to obtain the stimulation intensity of the area point; For each area point in the non-intersecting area, the position information of the area point and the posture information of the corresponding electrode wire and the stimulation parameter set are input into the single stimulation intensity model to obtain the stimulation intensity of the area point.

8. A wearable XR device, characterized in that The XR device includes: The nucleus marking device according to any one of claims 1 to 7; A camera, used to collect real-time images and send them to the nucleus marking device; The display screen is used to provide display functions.

9. A nucleus labeling method, characterized in that: The method comprises: Acquiring medical imaging data of a patient's brain, and reconstructing a three-dimensional model of the patient's brain based on the medical imaging data; Segmenting one or more nuclei from the medical image data to obtain a segmentation result of each nucleus; Acquire a first display image based on the segmentation result of each nucleus and the three-dimensional model, and display the first display image using a wearable XR device, wherein each nucleus is displayed separately in the first display image, and the position and size of each nucleus is marked for a doctor wearing the XR device; The acquiring of the first display image comprises: Capturing a real-time image through a camera of a wearable XR device, wherein the real-time image includes the patient's brain and one or more electrode leads implanted in the patient's brain; Registering the real-time image and the three-dimensional model to obtain a registration matrix; A first display image is acquired according to the segmentation result of each nuclear group, the registration matrix and the real-time image.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by at least one processor, implements the functions of the nucleus labeling device according to any one of claims 1 to 7, or implements the functions of the wearable XR device according to claim 8, or implements the steps of the method according to claim 9.

11. A computer program product, characterized in that The computer program product includes a computer program, which, when executed by at least one processor, implements the functions of the nuclear group labeling device described in any one of claims 1 to 7, or implements the functions of the wearable XR device described in claim 8, or implements the steps of the method described in claim 9.

Images