Brain nucleus detection method and device, electronic equipment and storage medium
By extracting visual features from brain images and embedding and fusing them with text, and using the YOLOv8 model to detect brain nuclei, the problem of low detection accuracy in existing technologies is solved, and high-precision nucleus detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2026-03-24
AI Technical Summary
Existing machine learning-based methods have low accuracy in detecting brain nuclei and cannot achieve high-precision detection.
By extracting visual features from brain images to obtain multi-scale feature maps, and combining them with a text encoder to convert the nucleus description information into vector representations, cross-modal fusion is performed before inputting the data into the YOLOv8 model for detection. Cosine similarity is used to determine the target embedding to achieve high-precision detection.
It improves the accuracy of brain nucleus detection and achieves high-precision detection of brain nuclei.
Smart Images
Figure CN119693291B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting brain nuclei. Background Technology
[0002] Brain nuclei are groups of neurons clustered in the subcortical region of the brain, including the substantia nigra, red nucleus, caudate nucleus, and amygdala. These nuclei play a crucial role in the functional and neural regulation of the brain.
[0003] In existing methods, the detection of brain nuclei generally relies on direct machine learning to segment brain images to extract brain nuclei. However, existing direct segmentation methods have low accuracy and cannot achieve high-precision detection of brain nuclei. Summary of the Invention
[0004] This invention provides a method, apparatus, electronic device, and storage medium for detecting brain nuclei, which addresses the shortcomings of existing machine learning-based methods for image segmentation of brain images, resulting in low accuracy, and achieves high-precision detection of brain nuclei.
[0005] This invention provides a method for detecting brain nuclei, comprising the following steps:
[0006] Visual features are extracted from the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
[0007] Based on a text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0008] Cross-modal fusion is performed on the multi-scale feature maps and the text embeddings to obtain fused features;
[0009] The fused features are input into the detection head of the YOLOv8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0010] Based on the similarity between each target embedding and the text embedding, the detection result of the target nucleus in the brain image to be detected is determined.
[0011] According to a brain nucleus detection method provided by the present invention, the step of determining the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding includes:
[0012] Based on the similarity between each target embedding and the text embedding, the target embedding with the highest similarity is determined;
[0013] The target embedding with the highest similarity and its corresponding predicted detection box are used as the detection result of the target nucleus in the brain image to be detected.
[0014] According to a brain nucleus detection method provided by the present invention, the step of determining the target embedding with the highest similarity based on the similarity between each target embedding and the text embedding includes:
[0015] Determine the cosine similarity between each target embedding and the text embedding;
[0016] The target embedding with the highest cosine similarity value is taken as the target embedding with the highest similarity.
[0017] According to a method for detecting brain nuclei provided by the present invention, after determining the detection result of the target nucleus in the brain image to be detected, the method further includes:
[0018] Based on the target embedding with the highest similarity and its corresponding predicted detection box, the brain nuclei in the brain image to be detected are located to determine the location information of the brain nuclei in the brain image to be detected.
[0019] According to a brain nucleus detection method provided by the present invention, the step of converting nucleus description information into vector representation based on a text encoder to obtain the text embedding of the nucleus description information includes:
[0020] A text encoder based on the CLIP model converts the nucleus description information into a vector representation to obtain the text embedding of the nucleus description information.
[0021] According to a method for detecting brain nuclei provided by the present invention, the step of extracting features from the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected includes:
[0022] Based on the backbone network in the YOLOv8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
[0023] The present invention also provides a brain nucleus detection device, comprising the following modules:
[0024] The image feature extraction module is used to extract visual features from the brain image to be detected, and obtain a multi-scale feature map of the brain image to be detected.
[0025] The text feature extraction module is used to convert the nucleus description information into a vector representation based on the text encoder to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0026] The feature fusion module is used to perform cross-modal fusion of the multi-scale feature map and the text embedding to obtain fused features;
[0027] The detection module is used to input the fused features into the detection head of the YOLOv8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0028] The comparison module is used to determine the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding.
[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the brain nucleus detection method as described above.
[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the brain nucleus detection method as described above.
[0031] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the brain nucleus detection method as described above.
[0032] The brain nucleus detection method, device, electronic device, and storage medium provided by this invention determine the multi-scale feature map of the brain image to be detected and the text embedding corresponding to the nucleus description information that characterizes the morphological and structural features of the brain nucleus. The multi-scale feature map and the text embedding are fused across modalities. At the same time, the brain nucleus is detected in real time using both brain image and text description data. The detection results are further filtered based on the text embedding, thereby achieving a high-precision detection process for brain nuclei and improving the detection accuracy. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0034] Figure 1 This is a flowchart illustrating the brain nucleus detection method provided by the present invention.
[0035] Figure 2 This is a schematic diagram of the brain nucleus detection device provided by the present invention.
[0036] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0038] Figure 1 This is a flowchart illustrating the brain nucleus detection method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:
[0039] Step 110: Visual feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
[0040] Step 120: Based on the text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of the target nucleus in the brain. The target nucleus includes the nucleus in the axial telencephalon and the nucleus in the brainstem.
[0041] Step 130: Perform cross-modal fusion on the multi-scale feature map and the text embedding to obtain fused features;
[0042] Step 140: Input the fused features into the detection head of the YOLOV8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0043] Step 150: Based on the similarity between each target embedding and the text embedding, determine the detection result of the target nucleus in the brain image to be detected.
[0044] The execution subject of the brain nucleus detection method provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), or personal computer (PC), etc. This invention does not impose specific limitations.
[0045] The following describes the technical solution of the present invention in detail using a computer executing the brain nucleus detection method provided by the present invention as an example.
[0046] In step 110, visual features are extracted from the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
[0047] Obtaining brain images. The brain images to be examined can be generated based on magnetic resonance imaging (MRI). MRI is a medical imaging technique that uses magnetic fields and radio frequency waves to obtain detailed anatomical structures and physiological information about the human body. Its non-invasive nature makes it highly suitable for brain imaging, and it can clearly display the brain's nuclei.
[0048] After acquiring the brain image to be tested, feature extraction is performed on the brain image to extract visual information and obtain a multi-scale feature map of the brain image to be tested.
[0049] Optionally, features can be extracted from the brain image to be detected based on a convolutional neural network, extracting multi-scale features. Convolutional neural networks have a multi-layer convolutional structure, which can effectively capture image features at different scales.
[0050] In step 120, the nucleus description information is converted into a vector representation based on a text encoder to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0051] Preliminary summarization of descriptive information about the target nuclei is crucial. It is understandable that brain nuclei possess a range of unique characteristics relative to other parts of the brain in imaging, giving them a significant role in the brain's structure and function. Furthermore, different brain nuclei exhibit distinct characteristics, including their structure, location, and signal signature.
[0052] Specifically, motor nuclei in the brain exhibit significant differences in size and shape. For example, the red nucleus is typically round and relatively large, while the substantia nigra is elongated and has a more extended structure. These characteristics are closely related to their respective functional properties. Therefore, descriptive information about target nuclei can be generated based on the unique morphological and structural features of brain nuclei relative to other parts of the brain image and the differences between brain nuclei.
[0053] The target nuclei include those in the axial telencephalon and those in the brainstem. The telencephalon nuclei are mainly gray matter nuclei, while the brainstem nuclei are distributed in the midbrain, pons, and medulla oblongata.
[0054] Choose a suitable text encoding model, such as the CLIP model, to transform the nucleus description information into a vector representation and generate a text embedding of the nucleus description information.
[0055] In step 130, the multi-scale feature map and the text embedding are fused across modally to obtain fused features.
[0056] After obtaining multi-scale feature maps and text embeddings, it is necessary to perform cross-modal fusion of the two in order to effectively combine image and text information.
[0057] Various fusion methods can be employed, such as weighted fusion, bilinear pooling, or attention mechanisms. For example, using an attention mechanism allows the model to dynamically focus on information from different modalities based on the needs of the task.
[0058] By using a cross-attention mechanism, text embeddings are used as queries, and image features are used as keys and values. Attention weights are calculated to obtain fused features, which can effectively capture the correlation between image features and text information.
[0059] In step 140, the fused features are input into the detection head of the YOLOV8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0060] The YOLOV8 model is pre-trained based on brain image samples and their corresponding brain nucleus labels to learn the mapping relationship between the features and labels of different nuclei.
[0061] Among them, the YOLOV8 (You Only Look Once) model is a real-time object detection algorithm that treats the object detection task as a single regression problem, directly mapping from image pixels to bounding box coordinates and class probabilities.
[0062] The fused features are input into the detection head of the YOLOv8 model. The detection head then performs a series of convolutional operations, outputting multiple object embeddings and their corresponding predicted detection boxes.
[0063] The object embedding output by the YOLOv8 detector head refers to encoding the features of objects in an image into a vector representation. Object embedding is a vectorized representation of the features of objects in an image. Its specific function is to extract the feature information of objects through the detector head so as to better classify, locate, and identify objects in subsequent processes.
[0064] In step 150, the detection results of the target nuclei in the brain image to be detected are determined based on the similarity between each target embedding and the text embedding.
[0065] The detection results of target nuclei in the brain image to be detected are determined by calculating the similarity between each target embedding and the text embedding. Specifically, this may include:
[0066] Similarity measurement: The cosine similarity method can be used to calculate the similarity between each target embedding and the text embedding.
[0067] The detection result of target nuclei in brain images is determined based on similarity values. From multiple target embeddings, the target embedding with the highest similarity to the text embedding is selected as the detection result for the target nucleus. This method effectively combines image features with text features, thereby improving the accuracy of nucleus detection.
[0068] The brain nucleus detection method provided by this invention determines the multi-scale feature map of the brain image to be detected and the text embedding corresponding to the nucleus description information that characterizes the morphological and structural features of the brain nucleus. The multi-scale feature map and the text embedding are fused across modalities. At the same time, the brain nucleus is detected in real time using both brain image and text description data. The detection results are further matched based on the text embedding, thereby achieving a high-precision detection process for the specified brain nucleus and improving the detection accuracy.
[0069] In one embodiment, determining the nucleus detection result of the brain image to be detected based on the similarity between each target embedding and the text embedding includes: determining the target embedding with the highest similarity based on the similarity between each target embedding and the text embedding; and using the target embedding with the highest similarity and its corresponding predicted detection box as the detection result of the target nucleus in the brain image to be detected.
[0070] After processing by the detection head of the YOLOv8 model, target embedding refers to the high-dimensional feature representation of each detected target (nucleus) in the image. These embeddings capture the visual features of the target in the image, including information such as shape, size, and texture.
[0071] Text embedding is a vector representation of descriptive information about brain nuclei, which is textual information used to characterize the morphology and function of brain nuclei. Text encoders (such as CLIP) transform this textual information into dense vectors with semantic meaning.
[0072] Similarity calculation is performed by measuring the degree of similarity between the target embedding and the text embedding. After similarity calculation, the best target embedding can be selected based on the similarity value. All target embeddings are iterated through to find the target embedding with the highest similarity to the text embedding.
[0073] After determining the target embedding with the highest similarity to the text embedding, the target embedding with the highest similarity and its corresponding predicted detection box are used as the detection result of the target nucleus in the brain image to be detected.
[0074] In one embodiment, determining the target embedding with the highest similarity based on the similarity between each target embedding and the text embedding includes: determining the cosine similarity between each target embedding and the text embedding; and selecting the target embedding with the largest cosine similarity value as the target embedding with the highest similarity.
[0075] Cosine similarity is a similarity calculation method based on a vector space model. It measures the similarity between two vectors by calculating the cosine of the angle between them. The closer the cosine of the angle between two vectors is to 1, the more similar they are; the closer it is to 0, the less similar they are.
[0076] For each target embedding, the cosine similarity between it and the text embedding is calculated. Based on the formula for calculating cosine similarity, a similarity score is obtained, which reflects the degree of similarity between the target embedding and the text embedding.
[0077] Among all the calculated cosine similarities, find the one with the largest value. The target embedding corresponding to this cosine similarity is the target embedding with the highest similarity.
[0078] In one embodiment, after determining the detection result of the target nucleus in the brain image to be detected, the method further includes: locating the brain nucleus in the brain image to be detected based on the target embedding with the highest similarity and its corresponding predicted detection box, so as to determine the location information of the brain nucleus in the brain image to be detected.
[0079] Understandably, the prediction detection box is typically a rectangular or elliptical region that surrounds the location information of the brain nucleus represented by the target embedding.
[0080] Using information from the predicted bounding box, the specific location of brain nuclei can be pinpointed within a brain image. This typically involves drawing a marker (such as a rectangle or ellipse) on the image corresponding to the predicted bounding box to visually indicate the location of the brain nuclei. After locating the brain nuclei, we can further extract their location information, such as coordinates, size, and shape.
[0081] In one embodiment, the text embedding of the nucleus description information is obtained by converting the nucleus description information into a vector representation based on a text encoder. This includes: a text encoder based on the CLIP model converting the nucleus description information into a vector representation to obtain the text embedding of the nucleus description information.
[0082] The CLIP text encoder is a crucial component of the CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is a multimodal pre-trained neural network that learns the alignment relationships between images and text through pre-training on a large amount of paired image and text data. The CLIP model has two main parts: a text encoder and an image encoder.
[0083] The CLIP-based text encoder transforms the nucleus description information into a vector representation, introducing complex semantic information and explicit instructions to obtain a text embedding for a specific nucleus detection task.
[0084] In one embodiment, feature extraction of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected includes: based on the backbone network in the YOLOV8 model, feature extraction of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
[0085] The YOLOv8 model is a real-time object detection algorithm whose core is to use a single neural network to directly generate bounding boxes and class probabilities from the input image. This model typically consists of two key parts: a backbone network and a detection head. The backbone network is responsible for extracting low-level features from the input image.
[0086] The backbone network performs a series of convolutional operations on the input brain image to be detected, extracting features at different scales. The feature map output by each convolutional layer corresponds to different image level information, such as subtle local changes, boundary features, and even higher-level abstract features, thus obtaining a multi-scale feature map of the brain image to be detected.
[0087] The brain nucleus detection device provided by the present invention is described below. The brain nucleus detection device described below can be referred to in correspondence with the brain nucleus detection method described above.
[0088] like Figure 2 As shown, the device includes:
[0089] Image feature extraction module 210 is used to extract visual features from the brain image to be detected, and obtain a multi-scale feature map of the brain image to be detected.
[0090] The text feature extraction module 220 is used to convert the nucleus description information into a vector representation based on the text encoder to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0091] Feature fusion module 230 is used to perform cross-modal fusion of the multi-scale feature map and the text embedding to obtain fused features;
[0092] Detection module 240 is used to input the fused features into the detection head of the YOLOV8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0093] The comparison module 250 is used to determine the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding.
[0094] The brain nucleus detection device provided by this invention determines the multi-scale feature map of the brain image to be detected and the text embedding corresponding to the nucleus description information that characterizes the morphological and structural features of the brain nucleus. It performs cross-modal fusion of the multi-scale feature map and the text embedding, and uses both brain image and text description data to perform real-time detection of brain nuclei. Based on the text embedding, it further matches the detection results to achieve a high-precision detection process for the specified brain nucleus and improve the detection accuracy.
[0095] In one embodiment, the comparison module 250 is specifically used for:
[0096] Based on the similarity between each target embedding and the text embedding, the detection results of the target nuclei in the brain image to be detected are determined, including:
[0097] Based on the similarity between each target embedding and the text embedding, the target embedding with the highest similarity is determined;
[0098] The target embedding with the highest similarity and its corresponding predicted detection box are used as the detection result of the target nucleus in the brain image to be detected.
[0099] In one embodiment, the comparison module 250 is further configured to:
[0100] Based on the similarity between each target embedding and the text embedding, the target embedding with the highest similarity is determined, including:
[0101] Determine the cosine similarity between each target embedding and the text embedding;
[0102] The target embedding with the highest cosine similarity value is taken as the target embedding with the highest similarity.
[0103] In one embodiment, the comparison module 250 is further configured to:
[0104] After determining the detection results of the target nucleus in the brain image to be detected, the method further includes:
[0105] Based on the target embedding with the highest similarity and its corresponding predicted detection box, the brain nuclei in the brain image to be detected are located to determine the location information of the brain nuclei in the brain image to be detected.
[0106] In one embodiment, the text feature extraction module 220 is specifically used for:
[0107] Based on a text encoder, the kernel description information is converted into a vector representation to obtain the text embedding of the kernel description information, including:
[0108] A text encoder based on the CLIP model converts the nucleus description information into a vector representation to obtain the text embedding of the nucleus description information.
[0109] In one embodiment, the image feature extraction module 210 is specifically used for:
[0110] Feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image, including:
[0111] Based on the backbone network in the YOLOV8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
[0112] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a brain nucleus detection method, which includes: extracting visual features from the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected;
[0113] Based on a text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0114] Cross-modal fusion is performed on the multi-scale feature maps and the text embeddings to obtain fused features;
[0115] The fused features are input into the detection head of the YOLOV8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0116] Based on the similarity between each target embedding and the text embedding, the detection result of the target nucleus in the brain image to be detected is determined.
[0117] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0118] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the brain nucleus detection method provided by the above methods, the method including: extracting visual features from the brain image to be detected, to obtain a multi-scale feature map of the brain image to be detected;
[0119] Based on a text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0120] Cross-modal fusion is performed on the multi-scale feature maps and the text embeddings to obtain fused features;
[0121] The fused features are input into the detection head of the YOLOV8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels.
[0122] Based on the similarity between each target embedding and the text embedding, the detection result of the target nucleus in the brain image to be detected is determined.
[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the brain nucleus detection method provided by the above methods, the method comprising: extracting visual features from a brain image to be detected to obtain a multi-scale feature map of the brain image to be detected;
[0124] Based on a text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information, which is a pre-determined descriptive information used to characterize the morphological and structural features of brain nuclei.
[0125] Cross-modal fusion is performed on the multi-scale feature maps and the text embeddings to obtain fused features;
[0126] The fusion features are input into the detection head of the YOLOV8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem.
[0127] Based on the similarity between each target embedding and the text embedding, the detection result of the target nucleus in the brain image to be detected is determined.
[0128] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for detecting brain nuclei, characterized in that, The method includes: Visual features are extracted from the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected. Based on a text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem. Based on the cross-attention mechanism, the text embedding is used as the query, the multi-scale feature map is used as the key and value, attention weights are calculated, and cross-modal fusion is performed on the multi-scale feature map and the text embedding based on the attention weights to obtain fused features; The fused features are input into the detection head of the YOLOv8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels. Based on the similarity between each target embedding and the text embedding, the target embedding with the highest similarity is determined; The target embedding with the highest similarity and its corresponding predicted detection box are used as the detection result of the target nucleus in the brain image to be detected.
2. The method for detecting brain nuclei according to claim 1, characterized in that, The step of determining the target embedding with the highest similarity based on the similarity between each target embedding and the text embedding includes: Determine the cosine similarity between each target embedding and the text embedding; The target embedding with the highest cosine similarity value is taken as the target embedding with the highest similarity.
3. The method for detecting brain nuclei according to claim 1, characterized in that, After determining the detection result of the target nucleus in the brain image to be detected, the method further includes: Based on the target embedding with the highest similarity and its corresponding predicted detection box, the brain nuclei in the brain image to be detected are located to determine the location information of the brain nuclei in the brain image to be detected.
4. The method for detecting brain nuclei according to claim 1, characterized in that, The step of converting the nucleus description information into a vector representation based on a text encoder to obtain the text embedding of the nucleus description information includes: A text encoder based on the CLIP model converts the nucleus description information into a vector representation to obtain the text embedding of the nucleus description information.
5. The method for detecting brain nuclei according to claim 1, characterized in that, The process of extracting features from the brain image to be detected, resulting in a multi-scale feature map of the brain image, includes: Based on the backbone network in the YOLOv8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.
6. A brain nucleus detection device, characterized in that, include: The image feature extraction module is used to extract visual features from the brain image to be detected, and obtain a multi-scale feature map of the brain image to be detected. The text feature extraction module is used to convert the nucleus description information into a vector representation based on the text encoder to obtain the text embedding of the nucleus description information. The nucleus description information is a pre-determined description information used to characterize the morphological and spatial features of target nuclei in the brain. The target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem. The feature fusion module is used to calculate attention weights based on the cross-attention mechanism, using the text embedding as a query and the multi-scale feature map as a key and value, and to perform cross-modal fusion of the multi-scale feature map and the text embedding based on the attention weights to obtain fused features. The detection module is used to input the fused features into the detection head of the YOLOv8 model to obtain all target embeddings and their corresponding predicted detection boxes output by the detection head. The YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels. The comparison module is used to determine the target embedding with the highest similarity based on the similarity between each target embedding and the text embedding; The target embedding with the highest similarity and its corresponding predicted detection box are used as the detection result of the target nucleus in the brain image to be detected.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the brain nucleus detection method as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the brain nucleus detection method as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the brain nucleus detection method as described in any one of claims 1 to 5.