Brain scan image recognition processing method, device, equipment and storage medium

Abstract

The application relates to a brain scan image recognition processing method, device and equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a brain scan image sample and midline annotation information; inputting the brain scan image sample into a first image recognition model to obtain a brain scan image sample corrected by registration output by a registration model branch and midline prediction information output by a recognition model branch; the registration model branch and the recognition model branch share part of a network in the first image recognition model; obtaining a loss function value based on the brain scan image sample corrected by registration, a standard brain scan image, the midline prediction information and the midline annotation information; and training the first image recognition model based on the loss function value, so that the influence of brain scan image registration on brain midline recognition is considered when training an end-to-end model for brain midline recognition, and the accuracy of brain midline recognition in the brain scan image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a brain scan image recognition processing method and device, equipment and a storage medium. BACKGROUND

[0002] In the medical field, the brain midline is an anatomical structure that separates the left and right hemispheres of the brain. How to quickly and accurately identify the brain midline plays an important role in clinical and scientific research fields.

[0003] In related technologies, the brain midline can be identified by a deep neural network. Specifically, two neural network models can be pre-trained, one is a registration model for registering an input brain scan image to a standard human brain, and the other is an identification model for processing the registered brain scan image to output the brain midline in the brain scan image.

[0004] However, in the above brain midline identification scheme, before identifying the brain midline of the brain scan image, it is necessary to first register it, and the registration process will lose part of the image information, resulting in low identification accuracy of the subsequent identification model. SUMMARY

[0005] The embodiments of the present application provide a brain scan image recognition processing method, device, equipment and storage medium, which can improve the identification accuracy of the model for the brain midline in the brain scan image. The technical scheme is as follows.

[0006] On the one hand, a brain scan image recognition processing method is provided, the method comprising:

[0007] obtaining a brain scan image sample and midline annotation information; the midline annotation information is used to indicate the brain midline annotated in the brain scan image sample;

[0008] inputting the brain scan image sample into a first image recognition model to obtain the brain scan image sample corrected by a registration model branch in the brain midline identification model, and to obtain midline prediction information output by an identification model branch in the brain midline identification model; the midline prediction information is used to indicate the brain midline predicted in the brain scan image sample; the registration model branch and the identification model branch share part of the network in the first image recognition model;

[0009] obtaining a loss function value based on the brain scan image sample corrected by the registration, a standard brain scan image, the midline prediction information and the midline annotation information;

[0010] training the first image recognition model based on the loss function value;

[0011] The first image recognition model after training is used to generate a second image recognition model, and the second image recognition model is used to process an input target brain scan image to output brain midline information of the target brain scan image, and the brain midline information is used to indicate a brain midline in the target brain scan image.

[0012] In another aspect, a method for recognizing a brain scan image is provided, and the method comprises:

[0013] obtaining a target brain scan image;

[0014] extracting features of the target brain scan image through a feature extraction branch in a second image recognition model to obtain image features;

[0015] processing the image features through a brain segmentation branch in the second image recognition model to obtain probability distribution information, and the probability distribution information is used to indicate a probability distribution of left and right hemispheres in the target brain scan image;

[0016] detecting a midline through a midline detection branch in the second image recognition model to obtain brain midline information, and the brain midline information is used to indicate a brain midline in the target brain scan image;

[0017] The second image recognition model is generated based on a first image recognition model after training; a loss function value of the first image recognition model is obtained based on a brain scan image sample after registration correction, a standard brain scan image, midline prediction information, and midline annotation information; the brain scan image sample after registration correction is output by a registration model branch in the brain midline recognition model, and the midline prediction information is output by a recognition model branch in the first image recognition model; the registration model branch and the recognition model branch share part of the network in the first image recognition model.

[0018] In another aspect, a device for recognizing a brain scan image is provided, and the device comprises:

[0019] a training data acquisition module configured to acquire a brain scan image sample and midline annotation information, and the midline annotation information is used to indicate a brain midline labeled in the brain scan image sample;

[0020] The model processing module is configured to input the brain scan image sample into a first image recognition model, obtain the brain scan image sample corrected by registration output by a registration model branch in the brain midline recognition model, and obtain midline prediction information output by an identification model branch in the brain midline recognition model. The midline prediction information is used to indicate a predicted brain midline in the brain scan image sample. The registration model branch and the identification model branch share part of the network in the first image recognition model.

[0021] The loss function obtaining module is configured to obtain a loss function value based on the brain scan image sample corrected by registration, a standard brain scan image, the midline prediction information, and the midline annotation information.

[0022] The training module is configured to train the first image recognition model based on the loss function value.

[0023] The first image recognition model after training is used to generate a second image recognition model. The second image recognition model is used to process a target brain scan image input by the user to output brain midline information of the target brain scan image. The brain midline information is used to indicate a brain midline in the target brain scan image.

[0024] In a possible implementation, the registration model branch includes a feature extraction branch and a registration branch, and the identification model branch includes the feature extraction branch, a brain segmentation branch, and a midline detection branch.

[0025] The model processing module is configured to,

[0026] The feature extraction branch is used to perform feature extraction on the brain scan image sample to obtain image sample features.

[0027] The registration branch is used to process the image sample features to obtain sample registration parameters.

[0028] The brain segmentation branch is used to process the image sample features to obtain prediction probability distribution information. The prediction probability distribution information is used to indicate a probability distribution of left and right hemispheres predicted in the brain scan image sample.

[0029] The brain scan image sample after removal of the skull is obtained based on the brain scan image sample and the prediction probability distribution information.

[0030] The brain scan image sample after removal of the skull is corrected based on the sample registration parameters to obtain the brain scan image sample corrected by registration.

[0031] The model processing module is configured to perform midline detection on the predicted probability distribution information by using the midline detection branch to obtain the midline prediction information.

[0032] In a possible implementation, when the model processing module performs midline detection on the predicted probability distribution information by using the midline detection branch to obtain the midline prediction information, the model processing module is configured to perform a three-dimensional convolution operation on a left brain binary image and a right brain binary image indicated by the predicted probability distribution information by using the midline detection branch to obtain the midline prediction information.

[0033] In a possible implementation, the loss function obtaining module includes:

[0034] A registration loss obtaining sub-module is configured to obtain a registration loss function value in the loss function value based on the brain scan image sample after registration correction and the standard brain scan image.

[0035] A segmentation loss obtaining sub-module is configured to obtain a segmentation loss function value in the loss function value based on the midline annotation information, the brain scan image sample, and the predicted probability distribution information.

[0036] A detection loss obtaining sub-module is configured to obtain a detection loss function value in the loss function value based on the midline prediction information and the midline annotation information.

[0037] In a possible implementation, the registration loss obtaining sub-module is configured to,

[0038] perform processing on the brain scan image sample after registration correction and the standard brain scan image by using a local normalized cross-correlation loss function to obtain a cross-correlation loss function value.

[0039] obtain the registration loss function value based on the cross-correlation loss function value.

[0040] In a possible implementation, in response to the registration manner being non-rigid registration, the sample registration parameter includes a three-dimensional deformation field.

[0041] In the process of obtaining the registration loss function value based on the cross-correlation loss function value, the registration loss obtaining sub-module is configured to,

[0042] perform an anisotropic deformation field smoothing loss process on the three-dimensional deformation field to obtain a registration smoothing loss function value.

[0043] obtain the registration loss function value based on the cross-correlation loss function value and the registration smoothing loss function value.

[0044] In a possible implementation, the segmentation loss obtaining sub-module is configured to,

[0045] obtain a weight map of the prediction probability distribution information based on the midline annotation information and the brain scan image sample;

[0046] perform cross-entropy loss calculation based on the weight map and the prediction probability distribution information to obtain the segmentation loss function value.

[0047] In a possible implementation, the detection loss obtaining sub-module is configured to,

[0048] perform loss calculation based on the midline prediction information and the midline annotation information to obtain a midline prediction loss function value;

[0049] perform smoothing loss calculation based on the midline prediction information to obtain a midline smoothing loss function value;

[0050] obtain the detection loss function value based on the midline prediction loss function value and the midline smoothing loss function value.

[0051] In a possible implementation, the training module is configured to,

[0052] update parameters of the feature extraction branch and the registration branch based on the registration loss function value;

[0053] update parameters of the feature extraction branch and the brain segmentation branch based on the segmentation loss function value;

[0054] update parameters of the midline detection branch based on the detection loss function value.

[0055] In a possible implementation, the apparatus further includes:

[0056] a model generating module configured to, in response to completion of training of the first image recognition model, generate the second image recognition model based on the feature extraction branch, the brain segmentation branch, and the midline detection branch.

[0057] In yet another aspect, an apparatus for recognizing a brain scan image is provided, and the apparatus includes:

[0058] an image obtaining module configured to obtain a target brain scan image;

[0059] a feature extraction module configured to perform feature extraction on the target brain scan image by a feature extraction branch in a second image recognition model to obtain image features;

[0060] a brain segmentation module configured to process the image features through a brain segmentation branch in the second image recognition model to obtain probability distribution information, the probability distribution information being used to indicate a probability distribution of left and right hemispheres in the target brain scan image;

[0061] a midline detection module configured to perform midline detection on the probability distribution information through a midline detection branch in the second image recognition model to obtain brain midline information, the brain midline information being used to indicate a brain midline in the target brain scan image;

[0062] In the second image recognition model is generated based on the first image recognition model after training; the loss function value of the first image recognition model is obtained based on the brain scan image sample after registration correction, the standard brain scan image, the midline prediction information and the midline annotation information; the brain scan image sample after registration correction is output by the registration model branch in the brain midline recognition model, and the midline prediction information is output by the recognition model branch in the first image recognition model; the registration model branch and the recognition model branch share part of the network in the first image recognition model.

[0063] In another aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the above-mentioned brain scan image recognition processing method.

[0064] In another aspect, a computer readable storage medium is provided, which stores at least one computer instruction, which is loaded and executed by a processor to implement the above-mentioned brain scan image recognition processing method.

[0065] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the above-mentioned brain scan image recognition processing method.

[0066] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0067] In the training process, the first image recognition model performs registration and brain midline recognition on the input brain scan image sample at the same time, and the first image recognition model is trained using the brain scan image sample corrected by registration, the standard brain scan image, the midline prediction information and the midline annotation information at the same time, that is, the influence of brain scan image registration on brain midline recognition is considered while training the end-to-end model for brain midline recognition, so that the end-to-end second image recognition model generated based on the trained first image recognition model can directly recognize the brain midline of the unregistered brain scan image, avoid the information loss caused by brain scan image registration, and improve the accuracy of recognizing the brain midline in the brain scan image.

[0068] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0069] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0070] Figure 1 is a system configuration diagram of a brain midline recognition system to which various embodiments of the present application relate;

[0071] Figure 2 is a flowchart of a brain scan image recognition processing method according to an exemplary embodiment;

[0072] Figure 3 is a flowchart of a brain scan image recognition processing method according to an exemplary embodiment;

[0073] Figure 4 is a brain scan image recognition processing framework diagram according to an exemplary embodiment;

[0074] Figure 5 is a flowchart of a brain scan image recognition processing method according to an exemplary embodiment;

[0075] Figure 6 is Figure 5 a model architecture diagram related to the embodiment shown;

[0076] Figure 7 is Figure 5 a schematic diagram of brain midline detection related to the embodiment shown;

[0077] Figure 8 is Figure 5 a distance-weight schematic diagram related to the embodiment shown;

[0078] Figure 9 is a training and application framework of a brain midline recognition model for human brain images according to an example embodiment;

[0079] Figure 10 is a structural block diagram of a brain scan image recognition processing device according to an example embodiment;

[0080] Figure 11 is a structural block diagram of a brain scan image recognition processing device according to an example embodiment;

[0081] Figure 12 is a structural schematic diagram of a computer device according to an example embodiment. DETAILED DESCRIPTION

[0082] The example embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, the same numbers refer to the same or similar elements throughout the drawings. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they only represent examples of apparatuses and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0083] Before the various embodiments shown in the present application are described, several concepts related to the present application are introduced.

[0084] 1) AI (Artificial Intelligence)

[0085] AI is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0086] 2) Computer Vision (CV)

[0087] Computer vision is a science that studies how to make machines "see". More specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics so that the computer processing becomes images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, and attempts to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0088] 3) ML (Machine Learning, Machine Learning)

[0089] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. It is applied in various fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0090] 4) Medical Cloud

[0091] Medical Cloud refers to the use of "cloud computing" to create a medical and health service cloud platform based on cloud computing, mobile technology, multimedia, wireless communication, big data, and Internet of Things, combined with medical technology, to realize the sharing of medical resources and the expansion of medical scope. Because of the combination of cloud computing technology, medical cloud improves the efficiency of medical institutions and facilitates residents to seek medical treatment. Like the current hospital's appointment registration, electronic medical records, medical insurance, etc. are the products of the combination of cloud computing and the medical field. Medical cloud also has the advantages of data security, information sharing, dynamic expansion, and global layout.

[0092] Please refer to Figure 1 , which shows a system structure diagram of a brain midline recognition system related to various embodiments of the present application. As Figure 1 indicated, the system includes a medical image acquisition device 120, a terminal 140, and a server 160; optionally, the system can also include a database 180.

[0093] The medical image acquisition device 120 can be a medical scanning device for acquiring brain scan images, such as a Computed Tomography (CT) device, a Magnetic Resonance Imaging (MRI) device, or the like. Correspondingly, the brain scan images refer to three-dimensional scanning images containing a brain, such as a brain CT image, a brain MRI image, or the like.

[0094] The medical image acquisition device 120 can include an image output interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, or an Ethernet interface, or the like; or the image output interface can also be a wireless interface, such as a Wireless Local Area Network (WLAN) interface, a Bluetooth interface, or the like.

[0095] Correspondingly, according to the type of the image output interface, the operation personnel can export the brain scan images in various ways, such as exporting the brain scan images to the terminal 140 through a wired or short-distance wireless manner, or exporting the brain scan images to the terminal 140 or the server 160 through a local area network or the Internet.

[0096] The terminal 140 can be a terminal device with certain processing capabilities and interface display functions, such as a mobile phone, a tablet computer, an e-book reader, smart glasses, a laptop computer, and a desktop computer, or the like.

[0097] The terminal 140 can include a terminal used by a developer and a terminal used by medical personnel.

[0098] When the terminal 140 is implemented as a terminal used by a developer, the developer can develop a machine learning model for identifying a brain midline in a brain scan image through the terminal 140, and deploy the machine learning model to the server 160 or a terminal used by medical personnel.

[0099] When the terminal 140 is implemented as a terminal used by medical personnel, the terminal 140 can be installed with an application program for obtaining and presenting a brain midline recognition result of a brain scan image. After the terminal 140 obtains the brain scan image acquired by the medical image acquisition device 120, the terminal 140 can obtain a processing result obtained by processing the brain scan image through the application program, and present the processing result, so as to facilitate a doctor to perform pathological diagnosis, surgical navigation, or the like.

[0100] exist Figure 1 In the system shown, terminal 140 and medical image acquisition device 120 are physically separate entities. Alternatively, in another possible implementation, when terminal 140 is implemented as a terminal for use by medical personnel, terminal 140 and medical image acquisition device 120 can also be integrated into a single entity; for example, terminal 140 can be a terminal device with medical three-dimensional image acquisition function.

[0101] Among them, server 160 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0102] The aforementioned server 160 may be a server that provides background services for applications installed in terminal 140. This background server may be responsible for application version management, background processing of brain scan images obtained by the application and returning the processing results, background training of machine learning models developed by developers, etc.

[0103] The aforementioned database 180 can be a Redis database, or it can be any other type of database. Database 180 is used to store various types of data.

[0104] Optionally, terminal 140 and server 160 are connected via a communication network. Optionally, medical image acquisition device 120 and server 160 are connected via a communication network. Optionally, this communication network can be a wired network or a wireless network.

[0105] Optionally, the system may also include a management device ( Figure 1 (Not shown), the management device is connected to the server 160 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0106] Figure 2 This is a flowchart illustrating a method for recognizing and processing brain scan images according to an exemplary embodiment. The method can be executed by a computer device, such as a server, a terminal, or a combination of both, wherein the server can be one of the aforementioned... Figure 1 The server 160 in the illustrated embodiment can be the terminal described above. Figure 1The terminal 140 used by the developer in the illustrated embodiment. The computer device can be implemented as a model training device for model training. As Figure 2 As shown, the brain scan image recognition processing method can include the following steps.

[0107] Step 201, obtaining a brain scan image sample and midline annotation information; the midline annotation information is used to indicate the brain midline annotated in the brain scan image sample.

[0108] Wherein, the brain scan image sample is a brain scan image that has been annotated with midline information in advance.

[0109] Wherein, in a single frame of brain scan image, the brain midline is a curve, and in a three-dimensional brain scan image, the brain midline is a curved surface.

[0110] Step 202, inputting the brain scan image sample into a first image recognition model to obtain a brain scan image sample corrected by registration output by a registration model branch in the brain midline recognition model, and obtaining midline prediction information output by an identification model branch in the brain midline recognition model; the midline prediction information is used to indicate the brain midline predicted in the brain scan image sample; the registration model branch and the identification model branch share part of the network in the first image recognition model.

[0111] In the embodiments of the present application, the first image recognition model can have two parallel output ports, and for the input brain scan image sample, the first image recognition model can output two results synchronously, one is the brain scan image sample after registration processing, and the other is the brain midline predicted from the original brain scan image sample.

[0112] Wherein, the above registration of the brain scan image sample refers to the operation of scaling and rotating the brain scan image sample to correct the size and direction of the brain scan image sample to the standard brain scan image.

[0113] In the embodiments of the present application, the above first image recognition model can include two branches, one of which is used to register the original brain scan image sample to obtain the brain scan image sample after registration correction, and the other branch is used to extract features from the original brain scan image sample and predict the brain midline according to the extracted features, that is, in the first image recognition model, the model part for predicting the brain midline is an end-to-end model branch.

[0114] Wherein, the above two branches can share part of the model network, such as sharing the model network of the feature extraction part.

[0115] In step 203, a loss function value is obtained based on the corrected brain scan image sample, the standard brain scan image, the midline prediction information, and the midline annotation information.

[0116] In step 204, the first image recognition model is trained based on the loss function value. The first image recognition model after training is used to generate a second image recognition model. The second image recognition model is used to process an input target brain scan image to output brain midline information of the target brain scan image. The brain midline information is used to indicate the brain midline in the target brain scan image.

[0117] In the embodiments of the present application, the second image recognition model described above is an end-to-end model, that is, the second image recognition model can extract features from the original target brain scan image and directly perform brain midline prediction based on the extracted features. In this process, the original target brain scan image does not need to be registered. Since the registration process is introduced in the model training process, and the registration process and the brain midline recognition process share part of the model network, the network of the brain midline recognition part can also learn the information related to image registration. Therefore, the end-to-end model obtained by training does not need to register the model, that is, it can achieve a similar effect as pre-registration. At the same time, the registration process will cause information loss, thereby improving the accuracy of the model in recognizing the brain midline.

[0118] In summary, in the training process, the first image recognition model simultaneously performs registration and brain midline recognition on the input brain scan image sample, and simultaneously uses the corrected brain scan image sample, the standard brain scan image, the midline prediction information, and the midline annotation information to train the first image recognition model. That is, while training the end-to-end model for brain midline recognition, the influence of brain scan image registration on brain midline recognition is also considered. Therefore, the end-to-end second image recognition model generated based on the trained first image recognition model can directly recognize the brain midline of the unregistered brain scan image, avoid information loss caused by brain scan image registration, and improve the accuracy of recognizing the brain midline in the brain scan image.

[0119] Based on the above Figure 2The first image recognition model trained by the scheme shown in the embodiment and the second image recognition model generated based on the first image recognition model can be applied to various scenarios of processing brain scan images and identifying the brain midline therein. For example, the second image processing model can be applied to identify the brain midline in a disease diagnosis assistance scenario to assist medical personnel in diagnosing the brain health status, or identify the brain midline in a brain surgery process to perform surgical navigation for the brain surgery. The process of the second image processing model for brain midline identification can refer to the following embodiments.

[0120] Figure 3 FIG. 1 is a flow diagram of a brain scan image identification processing method according to an example embodiment. The method can be performed by a computer device, for example, the computer device can be a server, or the computer device can also be a terminal, or the computer device can include a server and a terminal, wherein the server can be the server 160 in the embodiment shown above, and the terminal can be the terminal 140 used by medical personnel in the embodiment shown above. The computer device can be implemented as a model application device for brain midline identification. As shown in FIG. 1, the brain scan image identification processing method can include the following steps. Figure 1 Figure 1 The server 160 in the embodiment shown above, and the terminal 140 used by medical personnel in the embodiment shown above. The computer device can be implemented as a model application device for brain midline identification. As shown in FIG. 1, the brain scan image identification processing method can include the following steps. Figure 3

[0121] Step 301, obtaining a target brain scan image.

[0122] In the application process of brain midline identification, the computer device can obtain a target brain scan image selected or uploaded by a user.

[0123] Step 302, performing feature extraction on the target brain scan image through a feature extraction branch in a second image recognition model to obtain image features.

[0124] Step 303, processing the image features through a brain segmentation branch in the second image recognition model to obtain probability distribution information, the probability distribution information being used to indicate the probability distribution of left and right hemispheres in the target brain scan image.

[0125] Step 304, performing midline detection on the probability distribution information through a midline detection branch in the second image recognition model to obtain brain midline information, the brain midline information being used to indicate the brain midline in the target brain scan image.

[0126] ​​In the embodiment of the present application, the second image recognition model can be an end-to-end brain midline recognition model. The brain midline recognition model comprises a feature extraction branch, a brain segmentation branch and a midline detection branch connected in sequence, wherein the feature extraction branch is configured to perform feature extraction on the input target brain scan image, the brain segmentation branch is configured to perform left and right brain probability prediction based on the features of the target brain scan image, for example, to predict the probability of each pixel point in the target brain scan image belonging to the left brain, the right brain or the background (neither the left brain nor the right brain), and the midline detection branch is configured to output the brain midline after detecting the brain midline based on the probability prediction result.

[0127] The second image recognition model is generated based on the first image recognition model after training; the loss function value of the first image recognition model is obtained based on the brain scan image sample after registration correction, the standard brain scan image, the midline prediction information and the midline annotation information; the brain scan image sample after registration correction is output by the registration model branch in the brain midline recognition model, and the midline prediction information is output by the recognition model branch in the first image recognition model; the registration model branch and the recognition model branch share part of the network in the first image recognition model. The training process of the second image recognition model can refer to the description in the embodiments shown in Figure 2 The training process of the second image recognition model can refer to the description in the embodiments shown in

[0128] In summary, the scheme shown in the embodiments of the present application is that the second image recognition model is generated based on the first image recognition model, and in the training process of the first image recognition model, the first image recognition model simultaneously performs registration and brain midline recognition on the input brain scan image sample, and simultaneously uses the brain scan image sample after registration correction, the standard brain scan image, the midline prediction information and the midline annotation information to train the first image recognition model, that is, the end-to-end model for brain midline recognition is trained while considering the influence of brain scan image registration on brain midline recognition, so that the subsequent end-to-end second image recognition model generated based on the trained first image recognition model can directly recognize the brain midline of the unregistered brain scan image, avoid information loss caused by brain scan image registration, and thus improve the accuracy of recognizing the brain midline in the brain scan image.

[0129] The brain scan image recognition processing scheme involved in the present application can be divided into two stages, namely a model training stage and a model application stage. Please refer to Figure 4 , which is a brain scan image recognition processing framework diagram according to an exemplary embodiment. As Figure 4 shown, the brain scan image recognition processing process is as follows.

[0130] The computer device pre-acquires a brain scan image sample 401, midline annotation information 402, and a standard brain scan image 403. Among them, the midline annotation information 402 indicates the position of the brain midline in the brain scan image sample 401.

[0131] During the model training phase, the computer device inputs brain scan image samples 401 into the first image recognition model 404. The first image recognition model 404 performs registration and midline recognition on the brain scan image samples 401, and outputs a registration-corrected brain scan image sample 405 and a predicted midline information 406. The midline prediction information 406 indicates the position of the predicted midline in the brain scan image sample 401. Then, using the midline annotation information 402, the standard brain scan image 403, the registration-corrected brain scan image sample 405, and the predicted midline information 406, a loss function value is calculated, and the first image recognition model 404 is trained using this loss function value. This training process is repeated until the first image recognition model 404 is fully trained (e.g., reaching convergence).

[0132] After the first image recognition model 404 is trained, the computer device can automatically, or under the operation of the developer, generate a second image recognition model 407 based on the first image recognition model 404 and deploy the second image recognition model 407.

[0133] In the model application stage, the computer device inputs the target brain scan image 408 into the second image recognition model 407, and the second image recognition model 407 outputs brain midline information 409. Then, the computer device can output a recognition result image 410 that can indicate the brain midline in the target brain scan image based on the brain midline information 409, so that medical personnel can make corresponding decisions / judgments based on the recognition result image 410.

[0134] Figure 5 This is a flowchart illustrating a method for recognizing and processing brain scan images according to an exemplary embodiment. The method can be executed by a computer device, such as a server, a terminal, or a combination of both, wherein the server can be one of the aforementioned... Figure 1 The server 160 in the illustrated embodiment can be the terminal described above. Figure 1 Terminal 140 in the illustrated embodiment. (As shown) Figure 5 As shown, the brain scan image recognition and processing method may include the following steps.

[0135] At step 501, a brain scan image sample and midline annotation information are obtained. The midline annotation information is used to indicate a brain midline labeled in the brain scan image sample.

[0136] The brain scan image sample can be a brain scan image pre-collected by a model training personnel / developer, or can be a brain scan image in a public medical field dataset. The model training personnel / developer / professional medical personnel can label the brain scan image sample to determine the position of the brain midline in the brain scan image sample, to obtain the midline annotation information.

[0137] In a possible implementation, the midline annotation information can include the position of the brain midline in the brain scan image sample, for example, include the pixel coordinates of the brain midline in the brain scan image sample.

[0138] At step 502, the brain scan image sample is input into a first image recognition model to obtain a brain scan image sample corrected by a registration model branch of the brain midline recognition model, and obtain midline prediction information output by an identification model branch of the brain midline recognition model.

[0139] The midline prediction information is used to indicate a brain midline predicted in the brain scan image sample. The registration model branch and the identification model branch share part of the network in the first image recognition model.

[0140] After obtaining the brain scan image sample and the midline annotation information, the computer device can train the first image recognition model. In a possible implementation, the registration model branch of the first image recognition model can include a feature extraction branch and a registration branch, and the identification model branch of the first image recognition model can include the feature extraction branch, a brain segmentation branch, and a midline detection branch. That is, the registration model branch and the identification model branch can share the feature extraction branch.

[0141] The registration model branch and the identification model branch sharing the feature extraction branch can mean that the registration model branch and the identification model branch each have a feature extraction branch, and the structures and parameters of the two feature extraction branches are the same. Alternatively, the registration model branch and the identification model branch sharing the feature extraction branch can mean that the registration model branch and the identification model branch share the same feature extraction branch, that is, the features output by the feature extraction branch are input into the registration branch and the brain segmentation branch respectively.

[0142] The step 502 can include the following sub-steps:

[0143] At S502a, the brain scan image sample is feature-extracted by the feature extraction branch to obtain image sample features.

[0144] The feature extraction branch can include one or more convolutional layers, and is configured to perform feature extraction on the input brain scan image sample layer by layer. For example, a first convolutional layer in the feature extraction branch performs convolutional processing on the original data of the brain scan image sample to obtain a feature map output by the first convolutional layer. The feature map is then input to a second convolutional layer in the feature extraction branch (the feature map can be directly input to the second convolutional layer, or can be input to the second convolutional layer after being processed, for example, by pooling). The second convolutional layer processes the feature map to obtain a feature map output by the second convolutional layer. The feature map output by the second convolutional layer is further input to a third convolutional layer, and so on, until the last network layer of the feature extraction branch outputs the image sample feature.

[0145] S502b, processing the image sample feature through the registration branch to obtain a sample registration parameter.

[0146] In the embodiments of the present application, the registration branch can obtain a sample registration parameter used for registering the brain scan image sample by processing the image sample feature.

[0147] S502c, processing the image sample feature through the brain segmentation branch to obtain predicted probability distribution information, the predicted probability distribution information being used to indicate a probability distribution of a left and right brain predicted in the brain scan image sample.

[0148] The brain segmentation branch can also be a convolutional neural network including one or more convolutional layers. For example, the brain segmentation branch can process the input image sample feature through a plurality of convolutional layers in sequence, and finally output the predicted probability distribution information through an activation layer.

[0149] S502d, obtaining the brain scan image sample after the skull is removed based on the brain scan image sample and the predicted probability distribution information.

[0150] In the embodiments of the present application, the computer device can remove pixels in the brain scan image sample with a probability greater than a probability threshold from the background part to obtain the brain scan image sample after the skull is removed based on the predicted probability distribution information.

[0151] S502e, correcting the brain scan image sample after the skull is removed based on the sample registration parameter to obtain a registration-corrected brain scan image sample.

[0152] In the embodiments of the present application, the computer device can perform scaling, rotation, and the like on the brain scan image sample after the skull is removed based on the sample registration parameter to obtain the registration-corrected brain scan image sample.

[0153] S502f, the midline prediction information is obtained by performing midline detection on the predicted probability distribution information through the midline detection branch.

[0154] Wherein, the computer device can detect the brain midline of the brain scan image sample according to the probability of each pixel point in the brain scan image sample belonging to the left brain / right brain, to obtain the above-mentioned midline prediction information.

[0155] Please refer to Figure 6 , which shows a model architecture diagram related to an embodiment of the present application. In combination with Figure 6 , the model network of the first image recognition model related to the embodiment of the present application contains three parts. Among them, the feature extraction branch 601 part is the encoder of the feature, the input is the brain scan image sample 605, the brain segmentation branch 603 part is the feature decoder, which is used to output the probability map of the half brain segmentation 606, and the midline detection branch 604 is used to output the brain midline 607 through the probability map 606. The encoder and the decoder are connected through a jump connection result, which constitutes a segmentation network of a U-shaped network (U-Net). The input of the registration branch 602 part is the encoded feature, and the output is the deformation field or the affine matrix. Among them, when a non-rigid deformation is adopted, the registration branch 602 outputs a three-dimensional deformation field G ∈ R H×w×D×3 , and when a rigid deformation is adopted, the registration branch 602 outputs an affine matrix A ∈ R 3×4 , the input image (the original image input into the model, that is, 605) is changed (the brain image after skull removal is deformed rigidly / non-rigidly through a three-dimensional deformation field / affine matrix), and the corrected brain scan image 608 (i.e. the brain scan image sample after registration and correction) is output.

[0156] In a possible implementation, the midline prediction information is obtained by performing midline detection on the predicted probability distribution information through the midline detection branch, including:

[0157] The left brain binary graph and the right brain binary graph indicated by the predicted probability distribution information are subjected to a three-dimensional convolution operation through the midline detection branch, and the midline prediction information is obtained.

[0158] In the traditional algorithm, the process of generating the midline from the left and right brain probability is generally used as a post-processing option of the model, and is not integrated into the segmentation network, while in the present scheme, the midline prediction can be realized by a convolution network. For example, a three-dimensional (3-Dimensional, 3D) Sobel operator, K s The boundary gradient of the left and right brain is obtained. When executed, it can be realized by a three-dimensional convolution. The generation formula of the midline is as follows:

[0159] ml = Conv3d(brain l) *Conv3d(brain r )

[0160] Conv3d, full name is 3d convolution, that is, three-dimensional convolution operation, the convolution kernel here can be the convolution kernel of Sobel operator, brain l is the binary image of the left brain, brain r is the binary image of the right brain; the image gradient is also calculated by convolution operation, for example, the image gradient is calculated by Laplacian convolution kernel. Please refer to Figure 7 , which shows a schematic diagram of brain midline detection related to the embodiments of the present application.

[0161] The computer device can subsequently obtain a loss function value for training the first image recognition model based on the registration-corrected brain scan image sample, the standard brain scan image, the midline prediction information, and the midline annotation information.

[0162] In the embodiments of the present application, the loss function value for training the first image recognition model can include the loss of the registration part of the model, the loss of the brain segmentation part, and the loss of the midline detection part based on the brain segmentation result. The losses of the three parts can refer to steps 503 to 505 in the subsequent steps.

[0163] Step 503, based on the registration-corrected brain scan image sample and the standard brain scan image, obtaining a registration loss function value in the loss function value.

[0164] As Figure 6 shown, in the embodiments of the present application, for the corrected brain scan image 608, the computer device can calculate the loss function by the standard brain scan image 609 to obtain the loss in the image registration process, that is, the registration loss function value mentioned above.

[0165] In one possible implementation, the registration loss function value in the loss function value is obtained based on the registration-corrected brain scan image sample and the standard brain scan image, including:

[0166] The registration-corrected brain scan image sample and the standard brain scan image are processed by a local normalized cross-correlation loss function to obtain a cross-correlation loss function value;

[0167] Based on the cross-correlation loss function value, the registration loss function value is obtained.

[0168] In one possible implementation, in response to the registration mode being non-rigid registration, the sample registration parameters include a three-dimensional deformation field;

[0169] The registration loss function value is obtained based on the cross-correlation loss function value, and the registration loss function value comprises:

[0170] The anisotropic deformation field smoothing loss processing is performed on the three-dimensional deformation field to obtain a registration smoothing loss function value.

[0171] The registration loss function value is obtained based on the cross-correlation loss function value and the registration smoothing loss function value.

[0172] For example, in the embodiment of the present application, the computer device can add the cross-correlation loss function value and the registration smoothing loss function value to obtain the registration loss function value.

[0173] In the embodiment of the present application, the target function (loss function) of registration can be a local normalized cross-correlation loss (LNCC), and the formula is as follows:

[0174]

[0175] wherein, I and J are input images (I and J, one is the corrected brain scan image 608, and the other is the standard brain scan image 609); Ω is a set of all sliding windows; w is an element of Q, representing a local window; p is any point in the window w; I p is a pixel value of the P point; is the mean value of all pixels of the window w; is the local brightness difference of any point in the window, which is a 9x9x9 patch in the three-dimensional space; in the calculation, the sliding window method can be used to select the patch, and the convolution operation can make the calculation faster, wherein the smaller the NCC is, the more similar the images are.

[0176] In addition, if a non-rigid registration method is used, a smoothing constraint is further applied to the deformation field, so that the deformation is more smooth. The anisotropic deformation field smoothing loss is as follows:

[0177]

[0178] wherein and are anisotropic parameters for weighting the deformation along different axes. U is a three-dimensional deformation field U∈R HXWXDX3 , P is a local window, and U(P) is a patch with a size of WxWxW at the P point. The total registration loss is as follows:

[0179] l reg =-l lncc +l smooth (3)

[0180] At step 504, a segmentation loss function value in the loss function value is obtained based on the midline annotation information, the brain scan image sample, and the prediction probability distribution information.

[0181] In a possible implementation, the obtaining of the segmentation loss function value in the loss function value based on the midline annotation information, the brain scan image sample, and the prediction probability distribution information includes:

[0182] obtaining a weight map of the prediction probability distribution information based on the midline annotation information and the brain scan image sample;

[0183] performing cross-entropy loss calculation based on the weight map and the prediction probability distribution information to obtain the segmentation loss function value.

[0184] In the embodiments of the present application, the segmentation network can output a three-channel probability map, which is background, left brain, and right brain respectively. That is, in the above-mentioned probability map, there are three probabilities corresponding to each pixel point in the input image, which are the probability that the pixel point is background, the probability that the pixel point is left brain, and the probability that the pixel point is right brain. For the half-brain segmentation task, in an exemplary scheme, DistanceMap Weighted Cross Entropy Loss (DistanceMap Weighted Cross Entropy Loss) can be used for loss calculation. For example, first, the nearest distance of all points in the brain to the midline is calculated, and the points outside the brain are set to a distance of c=200mm. The distance-weighted diagram obtained can be as shown in FIG. 2. Figure 8

[0185] wherein the distance map needs to be converted into a weight map through the following formula for the weight of cross-entropy.

[0186] W=exp((c-D) / c) (4)

[0187]

[0188] In the above two formulas, c in formula 4 is a constant, which is 200mm here, that is, half the width of a human head, d is the distance of a point to the midline, and D represents a matrix composed of all distances.

[0189] In formula 5, c is a class, representing a category; N is the number of all points; g i is an indicator function, g i equals 1 when i=c, and otherwise equals 0. P i represents the probability value of category i; uppercase W is the weight matrix, and lowercase w represents the distance to a point. ​

[0190] In the embodiments of the present application, the distance map is introduced as a weight in the loss function, which can make the network pay attention to the centerline detection and improve the segmentation accuracy near the centerline. Compared with the traditional direct centerline segmentation method, the distance map is smoother, which can improve the training effect.

[0191] In step 505, the detection loss function value in the loss function value is obtained based on the centerline prediction information and the centerline annotation information.

[0192] In a possible implementation, the obtaining of the detection loss function value in the loss function value based on the centerline prediction information and the centerline annotation information includes:

[0193] loss calculation based on the centerline prediction information and the centerline annotation information to obtain a centerline prediction loss function value;

[0194] smooth loss calculation based on the centerline prediction information to obtain a centerline smooth loss function value;

[0195] obtaining the detection loss function value based on the centerline prediction loss function value and the centerline smooth loss function value.

[0196] After obtaining the segmentation result of the half brain, it is necessary to convert the half brain segmentation into centerline detection. In order to impose a surface smoothing constraint on the centerline, it is necessary to convert the binary segmentation probability map into centerline coordinates, as shown in Figure 6 The embodiments of the present application adopt a method similar to Argmax. Since Argmax cannot be differentiated in mathematics, an alternative scheme Soft-Argmax is adopted in the embodiments of the present application to calculate the centerline prediction loss function value, and the formula is as follows:

[0197]

[0198] In the above formula 6, i and j represent the coordinates of the current element in the one-dimensional vector, and β represents a constant. Since Soft-Argmax is similar to non-maximum suppression, it can enlarge the difference between the maximum value and the minimum value. X represents the weight, and xi represents the weight of the i th element.

[0199] In the loss function calculation process of the computer device, after obtaining the three-dimensional coordinates of the centerline, the surface smoothing constraint can be imposed. The surface constraint formula is as follows:

[0200]

[0201] Where u represents a surface (Surface) composed of all points. Surface can be represented by a HxW matrix, and each element is equivalent to the height of the corresponding point from the bottom surface.

[0202] To further improve the effect of midline detection, the embodiments of the present application also supervise the midline detection result. In the scheme, we use mean square error as the objective function. The formula is as follows:

[0203]

[0204] y and are the midline gold standard (i.e. midline annotation information) and the prediction result, respectively. Through the gold standard of the midline, the coordinates of each point not equal to 0 on the mask can be obtained.

[0205] Step 506, training the first image recognition model based on the loss function value.

[0206] In one possible implementation, the training of the first image recognition model based on the loss function value includes:

[0207] updating the parameters of the feature extraction branch and the registration branch based on the registration loss function value;

[0208] updating the parameters of the feature extraction branch and the brain segmentation branch based on the segmentation loss function value;

[0209] updating the parameters of the midline detection branch based on the detection loss function value.

[0210] Since the model network involved in the embodiments of the present application is a multi-task segmentation-registration network, the loss function of the network includes multiple items, and the overall loss function is as follows:

[0211] l = l surface + l region + l req + l smooth (9)

[0212] Wherein, the above l reg is used to train the registration branch part, l surface + l smooth is used to train the midline detection part, l region is used to train the half-brain segmentation part.

[0213] Step 507, in response to the completion of the training of the first image recognition model, generating the second image recognition model based on the feature extraction branch, the brain segmentation branch, and the midline detection branch.

[0214] Wherein, the second image recognition model is used to process the input target brain scan image to output the brain midline information of the target brain scan image, and the brain midline information is used to indicate the brain midline in the target brain scan image.

[0215] AsFigure 6 In the structure shown, in the inference application stage, the registration branch can be discarded, and only the identification model branch is retained for brain midline identification. Compared with running the first image identification model completely, the inference time can be significantly reduced.

[0216] In a possible implementation, the computer device can input the target brain scan image into the second image identification model to obtain position information of the brain midline output by the second image identification model; and based on the position information of the brain midline, output a brain midline identification image, where the brain midline identification image is used to indicate the brain midline in the target brain scan image. The process of processing the target brain scan image by the second image identification model to output the brain midline information is similar to the process of processing the brain scan image sample by the first image identification model to output the midline prediction information, and details are not described herein.

[0217] To sum up, in the training process of the first image identification model, the first image identification model simultaneously performs registration and brain midline identification on the input brain scan image sample, and simultaneously uses the brain scan image sample corrected by registration, the standard brain scan image, the midline prediction information, and the midline annotation information to train the first image identification model. That is, the end-to-end model for brain midline identification is trained while considering the influence of brain scan image registration on brain midline identification. Therefore, the subsequent end-to-end second image identification model generated based on the trained first image identification model can directly identify the brain midline from the unregistered brain scan image, avoid information loss caused by brain scan image registration, and improve the accuracy of identifying the brain midline in the brain scan image.

[0218] In the deep learning-based brain midline segmentation task, the scheme related to the above-mentioned embodiments of the present application adds a non-rigid or rigid registration method when designing the segmentation network, which registers the brain scan image to the standard human brain (in paired images, rigid registration refers to aligning one image to another template image through rotation, translation, and scaling), so that the network can learn the local deformation characteristics of the midline. The training network related to the above-mentioned embodiments of the present application combines the segmentation and registration tasks, which can be trained end-to-end to improve the midline segmentation effect. In offline inference, the registration branch can be pruned to achieve faster inference speed. In addition, in order to solve the problem of excessive local deformation during segmentation, the above-mentioned embodiments of the present application also design a three-dimensional curved surface smoothing loss as the objective function, so that the segmented midline is smoother. In addition, as the boundary line between the left and right hemispheres, the above-mentioned embodiments of the present application use a differentiable midline boundary detection module to convert the hemispheric segmentation result into midline segmentation. The entire module can be plug-and-play, and the brain midline recognition network model can be trained end-to-end. The scheme of the above-mentioned embodiments of the present application combines regional hemispheric segmentation and curved surface midline detection, which can better extract global symmetry features and local texture features.

[0219] As shown in the model structure of Figure 6 , the model network related to the above-mentioned embodiments of the present application mainly includes a segmentation network similar to U-Net, such as 3DU-Net, 3DFCN (Fully Convolution Network), etc., and a registration branch, wherein the registration branch is used to help the network learn the correction alignment of the brain scan image. During training, the model is trained in an end-to-end manner, and the model serves as a multi-task convolutional neural network to perform registration and brain midline detection tasks. During inference, in order to achieve faster speed, the network pruning method is used to remove the registration branch, and the model network only needs to output the brain midline detection result.

[0220] In clinical practice, three-dimensional brain midline has higher accuracy and is of great significance for measuring disease progression. The actual three-dimensional brain midline is a smooth curved surface in space. After applying the smoothness constraint of the curved surface, the brain midline recognition network related to the above-mentioned embodiments of the present application outputs better visual effects of the brain midline.

[0221] Please refer to Figure 9 , which is a training and application framework diagram of a brain midline recognition model for human brain images according to an exemplary embodiment. As shown in Figure 9 , the training and application process of the brain midline recognition model for human brain images can be as follows:

[0222] Firstly, the developer prepares a brain scan image sample 91a, and labels the brain scan image sample 91a to obtain midline annotation information 91b representing the real midline in the brain scan image sample 91a. In addition, the developer also prepares a standard brain scan image 91c for registration. Then, the brain scan image sample 91a, the midline annotation information 91b, and the standard brain scan image 91c are input into the model training device as training data.

[0223] In the model training phase, the model training device inputs the brain scan image sample 91a into the first image recognition model 92, processes it through the encoder in the first image recognition model 92 to obtain encoded features. The encoded features are decoded through the decoder in the first image recognition model 92 to obtain left and right brain prediction probability distribution information 94, and are processed through the registration branch in the first image recognition model 92 to obtain sample registration parameters 95. Subsequently, the prediction probability distribution information 94 is processed through the midline detection branch in the first image recognition model 92 to obtain midline prediction information 96. At the same time, the prediction probability distribution information 94 and the brain scan image sample 91a can be used to obtain a brain scan image sample 91d after removing the skull. The brain scan image sample 91d after removing the skull is corrected through the sample registration parameters 95 to obtain a registration-corrected brain scan image sample 91e.

[0224] The model training device can calculate a registration loss function value (including a registration smoothing loss when non-rigid registration) through the registration-corrected brain scan image sample 91e and the standard brain scan image 91c, and update the parameters of the encoder and the registration branch through the registration loss function value. The computer device can calculate a segmentation loss function value through the left and right brain indicated by the midline annotation information 91b and the prediction probability distribution information 94, and update the parameters of the encoder and the decoder through the segmentation loss function value. In addition, the computer device can also calculate a detection loss function value (optionally including a midline smoothing loss) through the midline prediction information 96 and the midline annotation information 91b, and update the parameters of the midline detection branch through the detection loss function value.

[0225] After the model training is completed, the developer can construct a second image recognition model 97 through the encoder, the decoder, and the midline detection branch in the first image recognition model 92, and deploy it to the model application device.

[0226] In the brain midline identification application stage, the model application device can receive an input target brain scan image 98, and input the target brain scan image 98 into the second image recognition model 97, output corresponding brain midline information 99 from the second image recognition model 97, and output a brain scan image 910 containing the identified brain midline based on the brain midline information 99.

[0227] In the above embodiments, the scheme can be implemented or executed in combination with a blockchain. For example, some or all steps in the above embodiments can be executed in a blockchain system; or data required for executing each step in the above embodiments or generated data can be stored in a blockchain system; for example, training samples used in the above model training and model input data such as target brain scan images in the model application process can be obtained by a computer device from a blockchain system; for example, parameters of the model obtained after the model training (including parameters of the first image recognition model and parameters of the second image recognition model) can be stored in a blockchain system.

[0228] Figure 10 is a structural block diagram of a brain scan image recognition processing device according to an exemplary embodiment. The device can implement Figure 2 or Figure 5 The brain scan image recognition processing device includes all or part of the steps in the method provided in the above embodiments.

[0229] The training data acquisition module 1001 is configured to acquire brain scan image samples and midline annotation information; the midline annotation information is used to indicate a brain midline annotated in the brain scan image samples;

[0230] The model processing module 1002 is configured to input the brain scan image samples into a first image recognition model, obtain a brain scan image sample corrected by a registration model branch in the brain midline recognition model, and obtain midline prediction information output by an identification model branch in the brain midline recognition model; the midline prediction information is used to indicate a brain midline predicted in the brain scan image samples; the registration model branch and the identification model branch share part of the network in the first image recognition model;

[0231] The loss function acquisition module 1003 is configured to acquire a loss function value based on the brain scan image sample corrected by the registration, a standard brain scan image, the midline prediction information, and the midline annotation information;

[0232] The training module 1004 is configured to train the first image recognition model based on the loss function value;

[0233] The first image recognition model after training is used to generate a second image recognition model, the second image recognition model is used to process an input target brain scan image to output brain midline information of the target brain scan image, and the brain midline information is used to indicate a brain midline in the target brain scan image.

[0234] In a possible implementation, the registration model branch includes a feature extraction branch and a registration branch, and the recognition model branch includes the feature extraction branch, a brain segmentation branch, and a midline detection branch.

[0235] The model processing module 1002 is configured to,

[0236] The feature extraction branch is used to perform feature extraction on the brain scan image sample to obtain image sample features.

[0237] The registration branch is used to process the image sample features to obtain sample registration parameters.

[0238] The brain segmentation branch is used to process the image sample features to obtain predicted probability distribution information, and the predicted probability distribution information is used to indicate a probability distribution of a left brain and a right brain predicted in the brain scan image sample.

[0239] Based on the brain scan image sample and the predicted probability distribution information, a skull-removed brain scan image sample is obtained.

[0240] The sample registration parameters are used to correct the skull-removed brain scan image sample to obtain a registration-corrected brain scan image sample.

[0241] The midline detection branch is used to perform midline detection on the predicted probability distribution information to obtain the midline prediction information.

[0242] In a possible implementation, when the midline detection branch is used to perform midline detection on the predicted probability distribution information to obtain the midline prediction information, the model processing module 1002 is configured to perform a three-dimensional convolution operation on a left brain binary image and a right brain binary image indicated by the predicted probability distribution information by using the midline detection branch to obtain the midline prediction information.

[0243] In a possible implementation, the loss function obtaining module 1003 includes:

[0244] The registration loss obtaining sub-module is configured to obtain a registration loss function value in the loss function value based on the registration-corrected brain scan image sample and the standard brain scan image.

[0245] The segmentation loss obtaining sub-module is configured to obtain a segmentation loss function value in the loss function value based on the midline annotation information, the brain scan image sample, and the prediction probability distribution information.

[0246] The detection loss obtaining sub-module is configured to obtain a detection loss function value in the loss function value based on the midline prediction information and the midline annotation information.

[0247] In a possible implementation, the registration loss obtaining sub-module is configured to,

[0248] The registration-corrected brain scan image sample and the standard brain scan image are processed by a local normalized cross-correlation loss function to obtain a cross-correlation loss function value.

[0249] The registration loss function value is obtained based on the cross-correlation loss function value.

[0250] In a possible implementation, in response to the registration manner being non-rigid registration, the sample registration parameter includes a three-dimensional deformation field.

[0251] When the registration loss function value is obtained based on the cross-correlation loss function value, the registration loss obtaining sub-module is configured to,

[0252] The three-dimensional deformation field is processed by an anisotropic deformation field smoothing loss to obtain a registration smoothing loss function value.

[0253] The registration loss function value is obtained based on the cross-correlation loss function value and the registration smoothing loss function value.

[0254] In a possible implementation, the segmentation loss obtaining sub-module is configured to,

[0255] The weight map of the prediction probability distribution information is obtained based on the midline annotation information and the brain scan image sample.

[0256] The segmentation loss function value is obtained by performing cross-entropy loss calculation based on the weight map and the prediction probability distribution information.

[0257] In a possible implementation, the detection loss obtaining sub-module is configured to,

[0258] The midline prediction loss function value is obtained by performing loss calculation based on the midline prediction information and the midline annotation information.

[0259] The midline smoothing loss function value is obtained by performing smoothing loss calculation based on the midline prediction information.

[0260] The detection loss function value is obtained based on the midline prediction loss function value and the midline smoothing loss function value.

[0261] In a possible implementation, the training module 1004 is configured to,

[0262] The parameters of the feature extraction branch and the registration branch are updated based on the registration loss function value;

[0263] The parameters of the feature extraction branch and the brain segmentation branch are updated based on the segmentation loss function value;

[0264] The parameters of the midline detection branch are updated based on the detection loss function value.

[0265] In a possible implementation, the apparatus further includes:

[0266] The model generation module is configured to, in response to the first image recognition model being trained, generate the second image recognition model based on the feature extraction branch, the brain segmentation branch, and the midline detection branch.

[0267] To sum up, in the training process, the first image recognition model simultaneously performs registration and brain midline recognition on the input brain scan image sample, and simultaneously uses the brain scan image sample corrected by registration, the standard brain scan image, the midline prediction information, and the midline annotation information to train the first image recognition model, that is, the end-to-end model for brain midline recognition is trained while the influence of brain scan image registration on brain midline recognition is considered, so that the end-to-end second image recognition model generated based on the trained first image recognition model can directly recognize the brain midline of the unregistered brain scan image, avoid information loss caused by brain scan image registration, and improve the accuracy of recognizing the brain midline in the brain scan image.

[0268] Figure 11 FIG. 1 is a structural block diagram of a brain scan image recognition processing apparatus according to an example embodiment. The brain scan image recognition processing apparatus can implement the method shown in the example embodiment. Figure 3 The brain scan image recognition processing apparatus includes all or part of the steps in the method shown in the example embodiment.

[0269] The image acquisition module 1101 is configured to acquire a target brain scan image.

[0270] The feature extraction module 1102 is configured to perform feature extraction on the target brain scan image by a feature extraction branch in the second image recognition model to obtain image features.

[0271] The brain segmentation module 1103 is configured to process the image feature by a brain segmentation branch in the second image recognition model to obtain probability distribution information, where the probability distribution information is used to indicate a probability distribution of left and right hemispheres in the target brain scan image.

[0272] The midline detection module 1104 is configured to perform midline detection on the probability distribution information by a midline detection branch in the second image recognition model to obtain brain midline information, where the brain midline information is used to indicate a brain midline in the target brain scan image.

[0273] In the method, the second image recognition model is generated based on a first image recognition model after training; a loss function value of the first image recognition model is obtained based on the brain scan image sample after registration correction, the standard brain scan image, the midline prediction information, and the midline annotation information; the brain scan image sample after registration correction is output by a registration model branch in the brain midline recognition model; the midline prediction information is output by a recognition model branch in the first image recognition model; and the registration model branch and the recognition model branch share part of a network in the first image recognition model.

[0274] In summary, the scheme shown in the embodiments of the present application is that the second image recognition model is generated based on the first image recognition model, and in the training process of the first image recognition model, the first image recognition model simultaneously performs registration and brain midline recognition on the input brain scan image sample, and simultaneously uses the brain scan image sample after registration correction, the standard brain scan image, the midline prediction information, and the midline annotation information to train the first image recognition model, that is, the influence of brain scan image registration on brain midline recognition is considered while training the end-to-end model for brain midline recognition, so that the end-to-end second image recognition model generated based on the first image recognition model after training can directly recognize the brain midline of the unregistered brain scan image, avoid information loss caused by brain scan image registration, and thus improve the accuracy of recognizing the brain midline in the brain scan image.

[0275] Figure 12is a structural schematic diagram of a computer device according to an exemplary embodiment. The computer device can be implemented as the computer device for training the first image recognition model in each of the above method embodiments, or can be implemented as the computer device for performing brain midline recognition through the second image recognition model in each of the above method embodiments. The computer device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The computer device 1200 also includes a basic input / output system (BIOS) 1206 that helps to transfer information between various devices in the computer, and a mass storage device 1207 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0276] The mass storage device 1207 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer readable media provide nonvolatile storage for the computer device 1200. That is, the mass storage device 1207 can include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0277] Without loss of generality, the computer readable medium can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, flash memory or other solid state memory technology, CD-ROM, or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media described above can not be limited to the above-mentioned several. The system memory 1204 and the mass storage device 1207 described above can be collectively referred to as memory.

[0278] The computer device 1200 can be connected to the Internet or other network devices through a network interface unit 1211 connected on the system bus 1205.

[0279] The memory also includes one or more programs stored therein, which one or more programs are executable by the central processing unit 1201 to implement all or part of the methods described above with regard to any of the methods illustrated in FIGS. 1-6. Figure 2 、 Figure 3 or Figure 5 any of the methods illustrated in FIGS. 1-6.

[0280] In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, which includes instructions, such as a memory including a computer program (instructions) executable by a processor of a computer device to perform the methods illustrated in the various embodiments of the present application. For example, the non-transitory computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0281] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods illustrated in the various embodiments described above.

[0282] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and that the claims be given their broadest interpretation and the fullest possible scope. The specification and examples are exemplary only and not intended to be limiting.

[0283] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is to be defined by the claims appended hereto.

Claims

1. A recognition processing method of a brain scan image, characterized by, The method comprises: obtaining a brain scan image sample and midline annotation information; the midline annotation information is used to indicate a brain midline annotated in the brain scan image sample; performing feature extraction on the brain scan image sample through a feature extraction branch in a first image recognition model to obtain image sample features; the first image recognition model comprises a registration model branch and an identification model branch; the registration model branch comprises the feature extraction branch and a registration branch; the identification model branch comprises the feature extraction branch, a brain segmentation branch and a midline detection branch; processing the image sample features through the registration branch to obtain sample registration parameters; processing the image sample features through the brain segmentation branch to obtain predicted probability distribution information, which is used to indicate a probability distribution of left and right hemispheres predicted in the brain scan image sample; based on the brain scan image sample and the predicted probability distribution information, obtaining the brain scan image sample after removing the skull; correcting the brain scan image sample after removing the skull based on the sample registration parameters to obtain the brain scan image sample after registration correction; performing midline detection on the predicted probability distribution information through the midline detection branch to obtain midline prediction information; the midline prediction information is used to indicate a brain midline predicted in the brain scan image sample; based on the brain scan image sample after registration correction, a standard brain scan image, the midline prediction information and the midline annotation information, obtaining a loss function value; training the first image recognition model based on the loss function value; the first image recognition model after training is used to generate a second image recognition model; the second image recognition model is used to process a target brain scan image input to output brain midline information of the target brain scan image; the brain midline information is used to indicate a brain midline in the target brain scan image.

2. The method of claim 1, wherein, The midline detection branch is used to perform three-dimensional convolution operation on the left brain binary image and the right brain binary image indicated by the predicted probability distribution information to obtain the midline prediction information. The loss function value is obtained based on the brain scan image sample after registration correction and the standard brain scan image, which comprises:

3. The method of claim 1, wherein, the registration loss function value in the loss function value is obtained based on the brain scan image sample after registration correction and the standard brain scan image; the segmentation loss function value in the loss function value is obtained based on the midline annotation information, the brain scan image sample and the predicted probability distribution information; the detection loss function value in the loss function value is obtained based on the midline prediction information and the midline annotation information. The registration loss function value in the loss function value is obtained based on the brain scan image sample after registration correction and the standard brain scan image, which comprises:

4. The method of claim 3, wherein, ​ processing the brain scan image sample and the standard brain scan image after registration correction by a local normalized cross-correlation loss function to obtain a cross-correlation loss function value; obtaining the registration loss function value based on the cross-correlation loss function value.

5. The method of claim 4, wherein, in response to the registration mode being non-rigid registration, the sample registration parameter comprises a three-dimensional deformation field; the obtaining of the registration loss function value based on the cross-correlation loss function value comprises: processing the three-dimensional deformation field by an anisotropic deformation field smoothing loss to obtain a registration smoothing loss function value; obtaining the registration loss function value based on the cross-correlation loss function value and the registration smoothing loss function value.

6. The method of claim 3, wherein, the obtaining of the segmentation loss function value in the loss function value based on the midline annotation information, the brain scan image sample and the prediction probability distribution information comprises: obtaining a weight map of the prediction probability distribution information based on the midline annotation information and the brain scan image sample; performing cross-entropy loss calculation based on the weight map and the prediction probability distribution information to obtain the segmentation loss function value.

7. The method of claim 3, wherein, the obtaining of the detection loss function value in the loss function value based on the midline prediction information and the midline annotation information comprises: performing loss calculation based on the midline prediction information and the midline annotation information to obtain a midline prediction loss function value; performing smoothing loss calculation based on the midline prediction information to obtain a midline smoothing loss function value; obtaining the detection loss function value based on the midline prediction loss function value and the midline smoothing loss function value.

8. The method of claim 3, wherein, the training of the first image recognition model based on the loss function value comprises: updating parameters of the feature extraction branch and the registration branch based on the registration loss function value; updating parameters of the feature extraction branch and the brain segmentation branch based on the segmentation loss function value; updating parameters of the midline detection branch based on the detection loss function value.

9. The method of claim 3, wherein, the method further comprises: in response to the training of the first image recognition model being completed, generating the second image recognition model based on the feature extraction branch, the brain segmentation branch and the midline detection branch.

10. A recognition processing method of a brain scan image, characterized by, the method comprises: obtaining a target brain scan image; performing feature extraction on the target brain scan image by a feature extraction branch in the second image recognition model to obtain image features; processing the image features by a brain segmentation branch in the second image recognition model to obtain probability distribution information, the probability distribution information being used to indicate probability distribution of left and right hemispheres in the target brain scan image; performing midline detection on the probability distribution information by a midline detection branch in the second image recognition model to obtain brain midline information, the brain midline information being used to indicate a brain midline in the target brain scan image; The second image recognition model is generated based on a trained first image recognition model; the first image recognition model comprises a registration model branch and an identification model branch, the registration model branch comprises a feature extraction branch and a registration branch, and the identification model branch comprises the feature extraction branch, a brain segmentation branch, and a midline detection branch; a loss function value of the first image recognition model is obtained based on a brain scan image sample after registration correction, a standard brain scan image, midline prediction information, and midline annotation information; the brain scan image sample after registration correction is obtained by performing feature extraction on the brain scan image sample through the feature extraction branch to obtain image sample features, performing processing on the image sample features through the registration branch to obtain sample registration parameters, performing processing on the image sample features through the brain segmentation branch to obtain prediction probability distribution information, obtaining the brain scan image sample after removal of the skull based on the brain scan image sample and the prediction probability distribution information, and correcting the brain scan image sample after removal of the skull based on the sample registration parameters; the prediction probability distribution information is used to indicate the probability distribution of the left and right hemispheres predicted in the brain scan image sample; and the midline prediction information is obtained by performing midline detection on the prediction probability distribution information through the midline detection branch.

11. A recognition processing apparatus of a brain scan image, characterized by comprising: The device comprises: a training data acquisition module configured to acquire a brain scan image sample and midline annotation information; the midline annotation information is used to indicate a brain midline annotated in the brain scan image sample; a model processing module configured to perform feature extraction on the brain scan image sample through a feature extraction branch in a first image recognition model to obtain image sample features; the first image recognition model comprises a registration model branch and an identification model branch, the registration model branch comprises the feature extraction branch and a registration branch, and the identification model branch comprises the feature extraction branch, a brain segmentation branch, and a midline detection branch; the model processing module is further configured to perform processing on the image sample features through the registration branch to obtain sample registration parameters, perform processing on the image sample features through the brain segmentation branch to obtain prediction probability distribution information, which is used to indicate the probability distribution of the left and right hemispheres predicted in the brain scan image sample, acquire the brain scan image sample after removal of the skull based on the brain scan image sample and the prediction probability distribution information, correct the brain scan image sample after removal of the skull based on the sample registration parameters to obtain the brain scan image sample after registration correction, and perform midline detection on the prediction probability distribution information through the midline detection branch to obtain midline prediction information, which is used to indicate a brain midline predicted in the brain scan image sample; a loss function acquisition module configured to obtain a loss function value based on the brain scan image sample after registration correction, a standard brain scan image, the midline prediction information, and the midline annotation information. The training module is configured to train the first image recognition model based on the loss function value. The first image recognition model after being trained is configured to generate a second image recognition model, and the second image recognition model is configured to process an input target brain scan image to output brain midline information of the target brain scan image, and the brain midline information is configured to indicate a brain midline in the target brain scan image.

12. A recognition processing apparatus of a brain scan image, characterized by comprising: The device comprises: An image acquisition module is configured to acquire a target brain scan image. A feature extraction module is configured to extract features of the target brain scan image by a feature extraction branch in the second image recognition model to obtain image features. A brain segmentation module is configured to process the image features by a brain segmentation branch in the second image recognition model to obtain probability distribution information, and the probability distribution information is configured to indicate probability distribution of left and right hemispheres in the target brain scan image. A midline detection module is configured to detect a brain midline by a midline detection branch in the second image recognition model to obtain brain midline information, and the brain midline information is configured to indicate the brain midline in the target brain scan image. The second image recognition model is generated based on the first image recognition model after being trained; the first image recognition model comprises a registration model branch and an identification model branch, the registration model branch comprises a feature extraction branch and a registration branch, and the identification model branch comprises the feature extraction branch, a brain segmentation branch, and a midline detection branch; a loss function value of the first image recognition model is obtained based on a brain scan image sample after registration correction, a standard brain scan image, midline prediction information, and midline annotation information; the brain scan image sample after registration correction is obtained by extracting features of the brain scan image sample by the feature extraction branch to obtain image sample features, processing the image sample features by the registration branch to obtain sample registration parameters, processing the image sample features by the brain segmentation branch to obtain predicted probability distribution information, obtaining the brain scan image sample after removing the skull based on the brain scan image sample and the predicted probability distribution information, and correcting the brain scan image sample after removing the skull based on the sample registration parameters; the predicted probability distribution information is configured to indicate predicted probability distribution of left and right hemispheres in the brain scan image sample; and the midline prediction information is obtained by detecting the brain midline from the predicted probability distribution information by the midline detection branch.

13. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the brain scan image recognition processing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer instruction, which is loaded and executed by the processor to implement the brain scan image recognition processing method according to any one of claims 1 to 10.

Images