Method for segmenting craniofacial ct images, method for training a model, and device
By combining semi-supervised deep neural networks and teacher networks, the problem of insufficient segmentation accuracy in craniofacial CT images in existing technologies is solved, achieving efficient image segmentation and diagnostic assistance while reducing the need for labeled data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI NINTH PEOPLES HOSPITAL SHANGHAI JIAO TONG UNIV SCHOOL OF MEDICINE
- Filing Date
- 2023-06-09
- Publication Date
- 2026-06-23
Smart Images
Figure CN116645377B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical imaging technology, and in particular to a method for segmenting craniofacial CT images, a method for training models, and an apparatus. Background Technology
[0002] Medical image segmentation is an important area in medical image processing, and its research has high practical and academic value. In recent years, with the continuous development of computer vision and machine learning technologies, significant progress has been made in medical image segmentation. Many deep learning-based image segmentation methods, such as U-Net, SegNet, and FCN, have been widely applied to medical image segmentation tasks. These methods, through the use of techniques such as convolutional neural networks (CNNs) and deconvolutional neural networks, have achieved high accuracy and stability in image segmentation tasks.
[0003] Despite significant progress in medical image segmentation, some shortcomings remain. On one hand, most existing medical image segmentation methods require large amounts of labeled data, which is a very difficult and time-consuming task, making it challenging to collect sufficient labeled data for model training. On the other hand, existing methods still struggle with complex medical images, such as those containing intricate structures, different types of tumors, or lesions. These factors impact segmentation accuracy, necessitating further research to address these issues.
[0004] Craniofacial tumors are common tumors that can affect any part of the craniofacial region. CT images are a common tool for diagnosing craniofacial tumors. By segmenting CT images, doctors can more accurately assess the size, shape, and location of the tumor, thereby determining the optimal treatment plan. However, craniofacial tumor CT images typically have high spatial complexity and uneven grayscale, and existing medical image segmentation networks are not specifically designed for the characteristics of craniofacial CT images. Summary of the Invention
[0005] In view of the problems in the prior art, the present invention provides a method for training a craniofacial CT image segmentation model, the method comprising:
[0006] A dataset of CT images of craniofacial tumors was obtained, which consisted of a labeled dataset and an unlabeled dataset.
[0007] The main network and the teacher network are constructed based on the same deep neural network backbone model for image segmentation.
[0008] The main network uses the labeled dataset to learn in each training iteration;
[0009] The teacher network learns using the labeled dataset in the preset start and end phases. In the intermediate phase, it simulates the learning process of the main network by averaging all the weights of the current main network. Furthermore, it constrains the graphical data in the unlabeled dataset by segmentation consistency from different perspectives.
[0010] Furthermore, the CT images of craniofacial tumors in the labeled dataset are labeled with tumor regions, maxillary regions, and mandibular regions.
[0011] Furthermore, the craniofacial tumor CT image dataset undergoes preprocessing to ensure the consistency of the individual image data.
[0012] Furthermore, the image segmentation deep neural network backbone model includes a convolutional part, a deconvolutional part, and a feature layer concatenation part. The feature layer concatenation part merges the features extracted by the convolutional part and the information recovered by the deconvolutional part.
[0013] Furthermore, the Dice similarity coefficient is used for evaluation on the test set, and the calculation formula is as follows:
[0014]
[0015] Among them, R GT For real image segmentation, R pre To predict image segmentation.
[0016] Furthermore, the different perspectives are based on three perspectives: transverse, coronal, and sagittal planes of CT images of craniofacial tumors.
[0017] Furthermore, the loss function used in the teacher network is:
[0018] L all =∑|A i -B i |+∑|A i -C i |+∑|C i -B i |
[0019] Among them, L all is the total loss value. A, B, and C represent the feature maps obtained from the same viewpoint after rotating the feature maps obtained by the teacher network from the cross-sectional, coronal, and sagittal views of the craniofacial tumor CT image. i represents the pixel index of the three-dimensional feature map.
[0020] The present invention also provides a method for segmenting craniofacial CT images, which uses a model trained by the above-mentioned craniofacial CT image segmentation model training method to segment craniofacial CT images.
[0021] The present invention also provides an electronic device, the electronic device comprising:
[0022] Processor; and
[0023] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the operations of the above-described method for training a craniofacial CT image segmentation model or a method for segmenting craniofacial CT images.
[0024] The present invention also provides a computer-readable medium storing instructions that, when executed, cause the system to perform the above-described method for training a craniofacial CT image segmentation model or a method for segmenting craniofacial CT images.
[0025] The present invention discloses a method for craniofacial CT image segmentation and a model training method. The model training method is based on a semi-supervised deep neural network-based craniofacial CT image segmentation algorithm. Addressing the difficulties in labeling craniofacial CT images and the problems of inaccurate segmentation of small objects and low consistency in 3D data segmentation in existing medical image segmentation networks, the present invention employs a deep neural network based on 3D convolutional modules to construct an encoding and decoding system, extracting tissue features from coarse to fine granular levels. Simultaneously, a semi-supervised structure based on a teacher network is used to train unlabeled data by using the segmentation consistency of the transverse, coronal, and sagittal views of the CT image as a supervision method, greatly alleviating the difficulty of 3D medical image labeling.
[0026] Therefore, the craniofacial CT image segmentation method and model training method of the present invention can be directly applied to the field of craniofacial CT image tissue segmentation, improving the accuracy and precision of craniofacial CT image segmentation, providing convenience and reliability in assisting doctors in diagnosing craniofacial diseases, and requiring no large amount of labeled data, greatly reducing the cost of manual labeling.
[0027] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0028] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0029] Figure 1 This is a schematic diagram of the architecture of one embodiment of the present invention;
[0030] Figure 2 The illustration shows a functional module of an exemplary system that can be used in various embodiments of the present invention.
[0031] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation
[0032] The present invention will now be described in further detail with reference to the accompanying drawings.
[0033] In a typical configuration of the present invention, the terminal, the device of the service network, and the trusted party all include one or more processors (e.g., a central processing unit (CPU)), input / output interfaces, network interfaces, and memory.
[0034] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory. Memory is an example of computer-readable media.
[0035] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information that can be accessed by a computing device.
[0036] The devices referred to in this invention include, but are not limited to, user equipment, network equipment, or devices composed of user equipment and network equipment integrated through a network. The user equipment includes, but is not limited to, any mobile electronic product capable of human-computer interaction (e.g., via a touchpad), such as smartphones and tablets. These mobile electronic products can use any operating system, such as Android or iOS. The network equipment includes an electronic device capable of automatically performing numerical calculations and information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices. The network equipment includes, but is not limited to, computers, network hosts, single network servers, multiple network server clusters, or clouds composed of multiple servers. Here, a cloud consists of a large number of computers or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual supercomputer composed of a group of loosely coupled computer clusters. The network includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, and wireless ad hoc network. Preferably, the device can also be a program running on the user equipment, network device, or a device formed by integrating user equipment and network device, network device, touch terminal, or network device and touch terminal through a network.
[0037] Of course, those skilled in the art should understand that the above-described devices are merely examples, and other existing or future devices that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.
[0038] In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0039] refer to Figure 1 One embodiment of the present invention provides a method for training a craniofacial CT image segmentation model, comprising:
[0040] S10. Obtain or prepare a certain number of craniofacial tumor CT image data as samples to form a craniofacial tumor CT image dataset. Select a portion of it as labeled data to form a labeled dataset, and the rest form an unlabeled dataset.
[0041] Preferably, labeled data and unlabeled data are grouped into batches in a 1:4 ratio for subsequent training.
[0042] For the labeled dataset, the corresponding tumor region, maxillary region, and mandibular region are labeled for each craniofacial tumor CT image. Based on subsequent training needs, the image data in the labeled dataset is divided into training set, test set, and validation set according to a predetermined ratio.
[0043] In some embodiments, to ensure the consistency of each image data in the craniofacial tumor CT image dataset, necessary preprocessing operations are performed on the image data in the craniofacial tumor CT image dataset. The preprocessing operations include image denoising, filtering, normalization, etc.
[0044] S20. Construct a deep neural network backbone model for image segmentation. The structure of the deep neural network backbone model mainly consists of a convolutional part, a deconvolutional part, and a feature layer concatenation part.
[0045] The convolutional part mainly consists of multiple convolutional layers and pooling layers to extract features from CT images of craniofacial tumors, such as tumors, maxilla, and mandible. Each convolutional layer convolves the image with a set of convolutional kernels to extract different features.
[0046] The deconvolution part is used to recover the information lost in the pooling layer back into the image. It mainly consists of multiple deconvolution layers, which recover the image information by learning the mapping from low-resolution image to high-resolution image.
[0047] The feature layer concatenation section combines the features extracted by the convolutional part and the information recovered by the deconvolutional part, thereby improving the accuracy of segmentation. The concatenation section consists of multiple concatenation layers, each of which combines the features from the two parts and passes them to the next layer.
[0048] By combining these components, the deep neural network backbone model can segment images by extracting features from the original image and then restoring it back to the image through deconvolution and stitching.
[0049] S30. Construct a semi-supervised neural network model, using the segmentation consistency of three perspective images (transverse, coronal, and sagittal) of craniofacial tumor CT images as the supervision method.
[0050] The semi-supervised neural network model includes a main network and a teacher network. The main network and the teacher network are constructed using the same network architecture. Specifically, the main network and the teacher network are constructed using the deep neural network backbone model in step S20.
[0051] The main network learns using a labeled dataset in each training iteration.
[0052] The teacher network learns using the labeled dataset only in the pre-defined start and end phases. In the intermediate phases, the teacher network simulates the learning process of the main network by averaging all the weights of the current main network.
[0053] In some embodiments, the averaging process uses an exponentially weighted average, as shown in the following formula:
[0054] P ema =β*P ema +(1-β)*p
[0055] Where p is the current parameter value, P ema It is the previous exponentially weighted average, and β is the decay coefficient. In some embodiments, β is 0.99.
[0056] Meanwhile, the teacher network constrains the segmentation consistency of graphic data in the unlabeled dataset through different perspectives. Specifically, it constrains the segmentation consistency of three perspectives: transverse, coronal, and sagittal, for CT images of craniofacial tumors.
[0057] In some embodiments, L1 loss is used as the loss function, and the specific loss function is expressed as follows:
[0058] L all =∑|A i -B i |+∑|A i -C i |+∑|C i -B i |
[0059] Among them, L all is the total loss value. A, B, and C represent the feature maps obtained from the same viewpoint after rotating the feature maps obtained through the teacher network in the transverse, coronal, and sagittal views of the CT image of craniofacial tumor. i represents the pixel index of the three-dimensional feature map.
[0060] S40. Input the batch of labeled and unlabeled data in proportion into the semi-supervised neural network model in step 30 for training.
[0061] In some embodiments, the Dice similarity coefficient is used for evaluation on the test set, and the calculation formula is as follows:
[0062]
[0063] Among them, R GT For real image segmentation, R pre To predict image segmentation.
[0064] The semi-supervised neural network model trained through steps S10 to S40 can be applied to actual craniofacial tumor CT image segmentation tasks to assist doctors in making diagnostic and treatment decisions.
[0065] One embodiment of the present invention provides a method for segmenting craniofacial CT images, which uses a semi-supervised neural network model trained by the operations described in steps S10 to S40 above to segment craniofacial CT images.
[0066] One embodiment of the present invention provides an electronic device, the electronic device comprising:
[0067] Processor; and
[0068] A memory is configured to store computer-executable instructions, which, when executed, cause the processor to perform the operation of the method for training the craniofacial CT image segmentation model in steps S10 to S40 above, or the operation of the method for segmenting craniofacial CT images described above.
[0069] An embodiment of the present invention provides a computer-readable medium storing instructions that, when executed, cause the system to perform the operation of the method for training the craniofacial CT image segmentation model in steps S10 to S40 above, or the operation of the method for segmenting craniofacial CT images described above.
[0070] Figure 2 Exemplary systems that can be used to implement the various embodiments described in this invention are shown.
[0071] like Figure 2 As shown, in some embodiments, system 1000 can function as any of the user terminal devices described in each of the embodiments. In some embodiments, system 1000 may include one or more computer-readable media having instructions (e.g., system memory or NVM / storage device 1020) and one or more processors (e.g., one or more processors 1005) coupled to the one or more computer-readable media and configured to execute the instructions to implement the module and thus perform the actions described in this invention.
[0072] In one embodiment, the system control module 1010 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1005 and / or any suitable device or component communicating with the system control module 1010.
[0073] The system control module 1010 may include a memory controller module 1030 to provide an interface to the system memory 1015. The memory controller module 1030 may be a hardware module, a software module, and / or a firmware module.
[0074] System memory 1015 may be used, for example, to load and store data and / or instructions for system 1000. In one embodiment, system memory 1015 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 1015 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).
[0075] In one embodiment, the system control module 1010 may include one or more input / output (I / O) controllers to provide interfaces to the NVM / storage device 1020 and (one or more) communication interfaces 1025.
[0076] For example, the NVM / storage device 1020 may be used to store data and / or instructions. The NVM / storage device 1020 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).
[0077] NVM / storage device 1020 may include storage resources that are physically part of a device on which system 1000 is mounted, or that can be accessed by the device without necessarily being part of the device. For example, NVM / storage device 1020 may be accessed via a network through one or more communication interfaces 1025.
[0078] One or more communication interfaces 1025 may provide the system 1000 with an interface to communicate over one or more networks and / or with any other suitable device. The system 1000 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and / or protocols.
[0079] In one embodiment, at least one of the processors 1005 may be logically packaged with one or more controllers of the system control module 1010 (e.g., memory controller module 1030). In one embodiment, at least one of the processors 1005 may be logically packaged with one or more controllers of the system control module 1010 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1005 may be integrated with the logic of one or more controllers of the system control module 1010 on the same die. In one embodiment, at least one of the processors 1005 may be integrated with the logic of one or more controllers of the system control module 1010 on the same die to form a system-on-a-chip (SoC).
[0080] In various embodiments, system 1000 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 1000 may have more or fewer components and / or different architectures. For example, in some embodiments, system 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.
[0081] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including associated data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of the present invention can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.
[0082] Furthermore, a portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0083] Communication media include media through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. Communication media can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided transmission) media capable of propagating energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as modulated data signals in, for example, wireless media (such as carrier waves or similar mechanisms embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.
[0084] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memories such as random access memory (RAM, DRAM, SRAM); and non-volatile memories such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or hereafter developed capable of storing computer-readable information / data for use by a computer system.
[0085] Here, according to one embodiment of the present invention, an apparatus is included, the apparatus including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run the methods and / or technical solutions based on the foregoing embodiments of the present invention.
[0086] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.
Claims
1. A method for training a segmentation model of craniofacial CT images, characterized in that, The method includes: A dataset of CT images of craniofacial tumors was obtained, which consisted of a labeled dataset and an unlabeled dataset. The main network and the teacher network are constructed based on the same deep neural network backbone model for image segmentation. The main network uses the labeled dataset to learn in each training iteration; The teacher network learns using the labeled dataset in the preset start and end phases. In the intermediate phase, it simulates the learning process of the main network by averaging all weights of the current main network. Furthermore, it constrains the segmentation consistency of the graphic data in the unlabeled dataset through different perspectives. These different perspectives are based on three views: transverse, coronal, and sagittal, from craniofacial tumor CT images. The loss function used by the teacher network is... in, That is the total loss value. A, B, and C represent feature maps obtained from the same viewpoint after rotating the feature maps obtained through the teacher network from the transverse, coronal, and sagittal views of a CT image of a craniofacial tumor. Represents the pixel index of the 3D feature map.
2. The method according to claim 1, characterized in that, The labeled dataset contains CT images of craniofacial tumors labeled with tumor regions, maxillary regions, and mandibular regions.
3. The method according to claim 1, characterized in that, The craniofacial tumor CT image dataset undergoes preprocessing to ensure consistency among the individual image data.
4. The method according to claim 1, characterized in that, The image segmentation deep neural network backbone model includes a convolutional part, a deconvolutional part, and a feature layer concatenation part. The feature layer concatenation part merges the features extracted by the convolutional part and the information recovered by the deconvolutional part.
5. The method according to claim 1, characterized in that, The Dice similarity coefficient was used for evaluation on the test set, and the calculation formula is as follows: in, For true image segmentation, To predict image segmentation.
6. A method for segmenting craniofacial CT images, characterized in that, Image segmentation is performed on craniofacial CT images using a model trained by the method described in any one of claims 1 to 5.
7. An electronic device, the electronic device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the operation of the method according to any one of claims 1 to 6.
8. A computer-readable medium storing instructions that, when executed, cause a system to perform operations according to any one of claims 1 to 6.