Brain nucleus detection method and device, electronic equipment and storage medium

Abstract

The invention provides a brain nucleus detection method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the visual feature extraction of a to-be-detected brain image, and obtaining a multi-scale feature map; on the basis of a text encoder, the kernel description information is converted into vector representation, and text embedding of the kernel description information is obtained; performing cross-modal fusion on the multi-scale feature map and text embedding to obtain fusion features; inputting the fusion features into a detection head of a YOLOv8 model to obtain all target embedding output by the detection head and a prediction detection frame corresponding to the target embedding; and determining a detection result of the target nuclei in the brain image to be detected based on the similarity between each target embedding and the text embedding. According to the method, the brain nucleuses are detected by means of two kinds of modal data of brain images and text description, and the detection results are further matched based on text embedding, so that the high-precision detection process of the specified brain nucleuses is realized, and the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a brain nucleus detection method, device, electronic equipment and storage medium. Background Art

[0002] Brain nuclei refer to some clustered neuronal groups under the cerebral cortex, including the substantia nigra, red nucleus, caudate nucleus, amygdala, etc. These nuclei play a vital role in the functional regulation and neural regulation of the brain.

[0003] In the existing methods, for the detection of brain nuclei, generally, the brain images are directly segmented based on machine learning methods to segment the brain nuclei images from the brain images. The existing direct segmentation methods have low accuracy and cannot achieve high-precision detection of brain nuclei. Summary of the invention

[0004] The present invention provides a brain nucleus detection method, device, electronic device and storage medium, which are used to solve the defect of low accuracy of image segmentation of brain images based on machine learning methods in the prior art, and realize high-precision detection of brain nuclei.

[0005] The present invention provides a method for detecting brain nuclei, comprising the following steps: Extract visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; Based on a text encoder, the nucleus description information is converted into a vector representation to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural characteristics and spatial characteristics of target nuclei in the brain, and the target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem; Performing cross-modal fusion on the multi-scale feature map and the text embedding to obtain fused features; Inputting the fusion features into the detection head of the YOLOv8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels; Based on the similarity between each target embedding and the text embedding, a detection result of the target nucleus in the brain image to be detected is determined.

[0006] According to a brain nucleus detection method provided by the present invention, the detection result of the target nucleus in the brain image to be detected is determined based on the similarity between each target embedding and the text embedding, including: Based on the similarity between each target embedding and the text embedding, determining the target embedding with the highest similarity; The target with the highest similarity is embedded and its corresponding predicted detection frame as the detection result of the target nucleus in the brain image to be detected.

[0007] According to a brain nucleus detection method provided by the present invention, the method of determining the target embedding with the highest similarity based on the similarity between each target embedding and the text embedding includes: Determining the cosine similarity of each target embedding with the text embedding; The target embedding with the largest cosine similarity value is used as the target embedding with the highest similarity.

[0008] According to a brain nucleus detection method provided by the present invention, after determining the detection result of the target nucleus in the brain image to be detected, the method further includes: Based on the target embedding with the highest similarity and its corresponding predicted detection frame, the brain nuclei in the brain image to be detected are located to determine the position information of the brain nuclei in the brain image to be detected.

[0009] According to a brain nucleus detection method provided by the present invention, the text encoder is used to convert the nucleus description information into a vector representation to obtain the text embedding of the nucleus description information, including: A text encoder based on the CLIP model converts the nucleus description information into a vector representation to obtain a text embedding of the nucleus description information.

[0010] According to a brain nucleus detection method provided by the present invention, the feature extraction of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected includes: Based on the backbone network in the YOLOv8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.

[0011] The present invention also provides a brain nucleus detection device, comprising the following modules: An image feature extraction module is used to extract visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; A text feature extraction module, used for converting the nucleus description information into a vector representation based on a text encoder to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural features and spatial features of target nuclei in the brain, and the target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem; A feature fusion module, used for cross-modal fusion of the multi-scale feature map and the text embedding to obtain fused features; A detection module, used for inputting the fusion features into a detection head of a YOLOv8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels; A comparison module is used to determine the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the brain nucleus detection methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the brain nucleus detection methods described above.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned brain nucleus detection methods.

[0015] The brain nucleus detection method, device, electronic device and storage medium provided by the present invention determine the text embedding corresponding to the multi-scale feature map of the brain image to be detected and the nucleus description information that characterizes the morphological and structural characteristics of the brain nuclei, and cross-modally fuse the multi-scale feature map and text embedding. At the same time, real-time detection of brain nuclei is performed with the help of two modal data, brain image and text description, and the detection results are further screened based on the text embedding, thereby realizing a high-precision detection process of brain nuclei and improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a schematic diagram of the flow of the brain nucleus detection method provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the structure of the brain nucleus detection device provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Figure 1 is a flow chart of the brain nucleus detection method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 110, extracting visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; Step 120, based on a text encoder, converting the nucleus description information into a vector representation to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural characteristics and spatial characteristics of a target nucleus in the brain, wherein the target nucleus includes the nucleus of the axial telencephalon and the nucleus in the brainstem; Step 130, cross-modal fusion is performed on the multi-scale feature map and the text embedding to obtain a fused feature; Step 140, inputting the fusion feature into the detection head of the YOLOV8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection boxes, wherein the YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels; Step 150, based on the similarity between each target embedding and the text embedding, determine the detection result of the target nucleus in the brain image to be detected.

[0022] The executor of the brain nucleus detection method provided by the present invention may be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), or a personal computer (PC), etc., which is not specifically limited by the present invention.

[0023] The following takes the brain nucleus detection method provided by the present invention executed by a computer as an example to explain the technical solution of the present invention in detail.

[0024] In step 110, visual features are extracted from the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.

[0025] Obtain brain images. The brain images to be tested can be generated based on magnetic resonance imaging (MRI). MRI is a medical imaging technology that uses magnetic fields and radio frequency waves to obtain detailed anatomical and physiological information inside the human body. It is very suitable for brain imaging because of its non-invasiveness and can clearly show the nucleus structure of the brain.

[0026] After obtaining the brain image to be detected, feature extraction is performed on the brain image to be detected, visual information of the brain image to be detected is extracted, and a multi-scale feature map of the brain image to be detected is obtained.

[0027] Optionally, feature extraction can be performed on the brain image to be detected based on a convolutional neural network to extract multi-scale features. The convolutional neural network has a multi-layer convolution structure and can effectively capture image features at different scales.

[0028] In step 120, based on a text encoder, the nuclear description information is converted into a vector representation to obtain a text embedding of the nuclear description information, wherein the nuclear description information is predetermined description information used to characterize the morphological structural characteristics and spatial characteristics of the target nuclei in the brain, and the target nuclei include the nuclei of the axial telencephalon and the nuclei in the brainstem.

[0029] Summarize and determine the target nucleus description information in advance. It is understandable that brain nuclei have a series of unique characteristics relative to other parts of the brain image, which make them play an important role in the structure and function of the brain. Different brain nuclei have different characteristics, including their structure, location, and signal characteristics.

[0030] Specifically, the motor nuclei in the brain nuclei have significant differences in size and shape. For example, the red nucleus is usually round and large in size. The substantia nigra is long and has a more extended structure. These characteristics are closely related to their respective functional characteristics. Therefore, the description information of the target nucleus can be generated based on the unique morphological and structural characteristics of the brain nuclei relative to other parts of the brain image and the different characteristics between the brain nuclei.

[0031] The target nuclei include those in the telencephalon in the axial plane and those in the brainstem. The nuclei in the telencephalon are mainly gray matter nuclei, while those in the brainstem are distributed in the midbrain, pons and medulla oblongata.

[0032] Select a suitable text encoding model, such as the CLIP model, to convert the nucleus description information into a vector representation and generate a text embedding of the nucleus description information.

[0033] In step 130, cross-modal fusion is performed on the multi-scale feature map and the text embedding to obtain a fused feature.

[0034] After obtaining the multi-scale feature maps and text embeddings, the two need to be fused cross-modally to effectively combine image and text information.

[0035] A variety of fusion methods can be used, such as weighted fusion, bilinear pooling, or attention mechanisms. For example, using attention mechanisms allows the model to dynamically focus on information from different modalities based on the needs of the task.

[0036] Through the cross-attention mechanism, text embedding is used as query, image features are used as keys and values, and attention weights are calculated to obtain fused features, which can effectively capture the correlation between image features and text information.

[0037] In step 140, the fusion feature is input into the detection head of the YOLOV8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection boxes. The YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels.

[0038] The YOLOV8 model is pre-trained based on brain image samples and their corresponding brain nucleus labels to learn the mapping relationship between the features and annotations of different nuclei.

[0039] Among them, the YOLOV8 (You Only Look Once) model is a real-time target detection algorithm that treats the target detection task as a single regression problem, directly mapping from image pixels to bounding box coordinates and category probabilities.

[0040] The fused features are input into the detection head of the YOLOV8 model. The detection head outputs multiple object embeddings and their corresponding predicted detection boxes through a series of convolution operations.

[0041] The object embedding output by the detection head of YOLOV8 refers to encoding the target features in the image into a vector representation. Object embedding is a vectorized representation of the features of objects in the image. Its specific function is to extract the feature information of the object through the detection head so as to better classify, locate and identify the object in the future.

[0042] In step 150, based on the similarity between each target embedding and the text embedding, the detection result of the target nucleus in the brain image to be detected is determined.

[0043] By calculating the similarity between each target embedding and the text embedding, the detection result of the target nucleus in the brain image to be detected is determined. Specifically, it may include: Similarity measure: The similarity between each target embedding and the text embedding can be calculated using the cosine similarity method.

[0044] The detection result of the target nucleus in the brain image is determined based on the similarity value. The target embedding with the highest similarity to the text embedding can be determined from multiple target embeddings as the detection result of the target nucleus. Through this method, the image features and text features can be effectively combined to improve the accuracy of nucleus detection.

[0045] The brain nucleus detection method provided by the present invention determines the multi-scale feature map of the brain image to be detected and the text embedding corresponding to the nucleus description information that characterizes the morphological and structural characteristics of the brain nuclei, and cross-modally fuses the multi-scale feature map and text embedding. At the same time, with the help of two modal data of brain image and text description, real-time detection of brain nuclei is performed, and the detection results are further matched based on the text embedding, thereby realizing a high-precision detection process for designated brain nuclei and improving the detection accuracy.

[0046] In one embodiment, based on the similarity between each target embedding and the text embedding, the nucleus detection result of the brain image to be detected is determined, including: based on the similarity between each target embedding and the text embedding, determining the target embedding with the highest similarity; using the target embedding with the highest similarity and its corresponding predicted detection box as the detection result of the target nucleus in the brain image to be detected.

[0047] After being processed by the detection head of the YOLOV8 model, object embedding refers to the high-dimensional feature representation of each object (nucleus) detected in the image. These embeddings capture the visual characteristics of the object in the image, including information such as shape, size, and texture.

[0048] Text embedding is a vector representation of the description information of the nuclei, which is text information used to characterize the morphology and function of brain nuclei. Text encoders (such as CLIP, etc.) convert this text information into dense vectors with semantic meaning.

[0049] The similarity calculation is performed by measuring the similarity between the target embedding and the text embedding. After the similarity calculation, the best target embedding can be selected based on the similarity value. Iterate through all target embeddings and find the target embedding with the highest similarity to the text embedding.

[0050] After determining the target embedding with the highest similarity to the text embedding, the target embedding with the highest similarity and its corresponding predicted detection box are used as the detection result of the target nucleus in the brain image to be detected.

[0051] In one embodiment, based on the similarity between each target embedding and the text embedding, determining the target embedding with the highest similarity includes: determining the cosine similarity between each target embedding and the text embedding; and taking the target embedding with the largest cosine similarity value as the target embedding with the highest similarity.

[0052] Cosine similarity is a similarity calculation method based on the vector space model. It measures the similarity between two vectors by calculating the cosine value of the angle between them. The closer the cosine value of the angle between two vectors is to 1, the more similar they are; the closer it is to 0, the less similar they are.

[0053] For each target embedding, calculate the cosine similarity between it and the text embedding. According to the calculation formula of cosine similarity, a similarity score can be obtained, which reflects the similarity between the target embedding and the text embedding.

[0054] Among all the calculated cosine similarities, find the one with the largest value. The target embedding corresponding to this cosine similarity is the target embedding with the highest similarity.

[0055] In one embodiment, after determining the detection result of the target nucleus in the brain image to be detected, it also includes: based on the target embedding with the highest similarity and its corresponding predicted detection frame, locating the brain nucleus in the brain image to be detected to determine the position information of the brain nucleus in the brain image to be detected.

[0056] It is understandable that the predicted detection box is usually a rectangular or elliptical area that surrounds the location information of the brain nucleus represented by the target embedding.

[0057] Using the information of the predicted detection box, the specific location of the brain nucleus can be located in the brain image to be detected. This usually involves drawing a mark (such as a rectangular box, an elliptical box, etc.) corresponding to the predicted detection box on the image to intuitively display the location of the brain nucleus. After locating the brain nucleus, we can further extract its location information, such as coordinates, size, shape, etc.

[0058] In one embodiment, based on a text encoder, the nucleus description information is converted into a vector representation to obtain a text embedding of the nucleus description information, including: a text encoder based on a CLIP model, converting the nucleus description information into a vector representation to obtain a text embedding of the nucleus description information.

[0059] CLIP's text encoder is an important part of the CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is a multimodal pre-trained neural network that is pre-trained with a large amount of image and text pairing data to learn the alignment relationship between images and text. The CLIP model has two main parts: the text encoder and the image encoder.

[0060] A text encoder based on the CLIP model converts the nucleus description information into a vector representation, introduces complex semantic information and explicit instructions, and obtains text embedding for a specific nucleus detection task.

[0061] In one embodiment, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected, including: based on the backbone network in the YOLOV8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.

[0062] The YOLOV8 model is a real-time object detection algorithm, the core of which is to use a single neural network to generate bounding boxes and category probabilities directly from the input image. The model usually consists of two key parts: the backbone network and the detection head. The backbone network is responsible for extracting low-level features from the input image.

[0063] The backbone network performs a series of convolution operations on the input brain image to be tested to extract features at different scales. The feature map output by each convolution layer corresponds to different image level information, such as subtle local changes, boundary features, and even higher-level abstract features, thereby obtaining a multi-scale feature map of the brain image to be tested.

[0064] The brain nucleus detection device provided by the present invention is described below. The brain nucleus detection device described below and the brain nucleus detection method described above can be referenced to each other.

[0065] like Figure 2 As shown, the device comprises: An image feature extraction module 210 is used to extract visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; A text feature extraction module 220, for converting the nucleus description information into a vector representation based on a text encoder to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural features and spatial features of target nuclei in the brain, wherein the target nuclei include the nuclei in the axial telencephalon and the nuclei in the brainstem; A feature fusion module 230, configured to perform cross-modal fusion on the multi-scale feature map and the text embedding to obtain a fused feature; A detection module 240 is used to input the fusion feature into a detection head of a YOLOV8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection boxes, wherein the YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels; The comparison module 250 is used to determine the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding.

[0066] The brain nucleus detection device provided by the present invention determines the multi-scale feature map of the brain image to be detected and the text embedding corresponding to the nucleus description information that characterizes the morphological and structural characteristics of the brain nuclei, and cross-modally fuses the multi-scale feature map and text embedding. At the same time, with the help of two modal data of brain image and text description, real-time detection of brain nuclei is performed, and the detection results are further matched based on the text embedding, thereby realizing a high-precision detection process for designated brain nuclei and improving the detection accuracy.

[0067] In one embodiment, the comparison module 250 is specifically used for: Determining the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding, including: Based on the similarity between each target embedding and the text embedding, determining the target embedding with the highest similarity; The target with the highest similarity is embedded and its corresponding predicted detection frame as the detection result of the target nucleus in the brain image to be detected.

[0068] In one embodiment, the comparison module 250 is further specifically configured to: Based on the similarity between each target embedding and the text embedding, determining the target embedding with the highest similarity includes: Determining the cosine similarity of each target embedding with the text embedding; The target embedding with the largest cosine similarity value is used as the target embedding with the highest similarity.

[0069] In one embodiment, the comparison module 250 is further specifically configured to: After determining the detection result of the target nucleus in the brain image to be detected, the method further includes: Based on the target embedding with the highest similarity and its corresponding predicted detection frame, the brain nuclei in the brain image to be detected are located to determine the position information of the brain nuclei in the brain image to be detected.

[0070] In one embodiment, the text feature extraction module 220 is specifically used for: Based on the text encoder, the nucleus description information is converted into a vector representation to obtain the text embedding of the nucleus description information, including: A text encoder based on the CLIP model converts the nucleus description information into a vector representation to obtain a text embedding of the nucleus description information.

[0071] In one embodiment, the image feature extraction module 210 is specifically used for: Performing feature extraction on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected, including: Based on the backbone network in the YOLOV8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.

[0072] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the brain nucleus detection method, which includes: extracting visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; Based on a text encoder, the nucleus description information is converted into a vector representation to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural characteristics and spatial characteristics of target nuclei in the brain, and the target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem; Performing cross-modal fusion on the multi-scale feature map and the text embedding to obtain fused features; Inputting the fusion feature into the detection head of the YOLOV8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels; Based on the similarity between each target embedding and the text embedding, a detection result of the target nucleus in the brain image to be detected is determined.

[0073] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0074] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the brain nucleus detection method provided by the above methods, the method comprising: extracting visual features of a brain image to be detected, and obtaining a multi-scale feature map of the brain image to be detected; Based on a text encoder, the nucleus description information is converted into a vector representation to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural characteristics and spatial characteristics of target nuclei in the brain, and the target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem; Performing cross-modal fusion on the multi-scale feature map and the text embedding to obtain fused features; Inputting the fusion feature into the detection head of the YOLOV8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the YOLOV8 model is trained based on brain image samples and their corresponding brain nucleus labels; Based on the similarity between each target embedding and the text embedding, a detection result of the target nucleus in the brain image to be detected is determined.

[0075] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the brain nucleus detection method provided by the above methods, the method comprising: extracting visual features of a brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; Based on a text encoder, the nucleus description information is converted into a vector representation to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information used to characterize morphological structural features of brain nuclei; Performing cross-modal fusion on the multi-scale feature map and the text embedding to obtain fused features; Input the fusion feature into the detection head of the YOLOV8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the nucleus description information is predetermined description information for characterizing the morphological structural features and spatial features of the target nuclei in the brain, and the target nuclei include the nuclei in the axial telencephalon and the nuclei in the brainstem; Based on the similarity between each target embedding and the text embedding, a detection result of the target nucleus in the brain image to be detected is determined.

[0076] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting brain nuclei, characterized in that: The method comprises: Extract visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; Based on a text encoder, the nucleus description information is converted into a vector representation to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural characteristics and spatial characteristics of target nuclei in the brain, and the target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem; Performing cross-modal fusion on the multi-scale feature map and the text embedding to obtain fused features; Inputting the fusion features into the detection head of the YOLOv8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels; Based on the similarity between each target embedding and the text embedding, a detection result of the target nucleus in the brain image to be detected is determined.

2. The brain nucleus detection method according to claim 1, characterized in that: The step of determining the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding includes: Based on the similarity between each target embedding and the text embedding, determining the target embedding with the highest similarity; The target with the highest similarity is embedded and its corresponding predicted detection frame as the detection result of the target nucleus in the brain image to be detected.

3. The brain nucleus detection method according to claim 2, characterized in that: The determining, based on the similarity between each target embedding and the text embedding, the target embedding with the highest similarity comprises: Determining the cosine similarity of each target embedding with the text embedding; The target embedding with the largest cosine similarity value is used as the target embedding with the highest similarity.

4. The brain nucleus detection method according to claim 2, characterized in that: After determining the detection result of the target nucleus in the brain image to be detected, the method further includes: Based on the target embedding with the highest similarity and its corresponding predicted detection frame, the brain nuclei in the brain image to be detected are located to determine the position information of the brain nuclei in the brain image to be detected.

5. The brain nucleus detection method according to claim 1, characterized in that: The text encoder-based method converts the nucleus description information into a vector representation to obtain a text embedding of the nucleus description information, including: A text encoder based on the CLIP model converts the nucleus description information into a vector representation to obtain a text embedding of the nucleus description information.

6. The brain nucleus detection method according to claim 1, characterized in that: The feature extraction of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected includes: Based on the backbone network in the YOLOv8 model, feature extraction is performed on the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected.

7. A brain nucleus detection device, characterized in that: include: An image feature extraction module is used to extract visual features of the brain image to be detected to obtain a multi-scale feature map of the brain image to be detected; A text feature extraction module, used for converting the nucleus description information into a vector representation based on a text encoder to obtain a text embedding of the nucleus description information, wherein the nucleus description information is predetermined description information for characterizing the morphological structural features and spatial features of target nuclei in the brain, and the target nuclei include nuclei in the axial telencephalon and nuclei in the brainstem; A feature fusion module, used for cross-modal fusion of the multi-scale feature map and the text embedding to obtain fused features; A detection module, used for inputting the fusion features into a detection head of a YOLOv8 model to obtain all target embeddings output by the detection head and their corresponding predicted detection frames, wherein the YOLOv8 model is trained based on brain image samples and their corresponding brain nucleus labels; A comparison module is used to determine the detection result of the target nucleus in the brain image to be detected based on the similarity between each target embedding and the text embedding.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the brain nucleus detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the brain nucleus detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the brain nucleus detection method according to any one of claims 1 to 6 is implemented.

Images