A CBCT image tooth segmentation method based on Mamba and multi-modal fusion

Abstract

This invention provides a tooth segmentation method for CBCT images based on Mamba and multimodal fusion, relating to the fields of medical image processing and computer vision. This invention constructs a multi-channel input tensor that fuses and enhances jawbone images and spatial coordinate maps, explicitly providing the network with voxel absolute position priors. It also strengthens the structural boundaries of low-contrast regions such as the junction of tooth roots and alveolar bone using a spatially adaptive contrast enhancement strategy, mitigating tissue confusion caused by similar gray levels. By dynamically injecting diagnostic semantic priors from case texts into image bottleneck features based on spatial importance, it effectively suppresses category confusion between morphologically similar symmetrical teeth with the same name. Simultaneously, it utilizes the Mamba state space model to perform multi-directional sequence dependency modeling and omnidirectional aggregation of multimodal fusion features, efficiently capturing the ultra-long-range three-dimensional spatial dependencies of teeth along the arch curvature with near-linear computational complexity, achieving a balance between global context modeling capability and computational efficiency.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, specifically to a tooth segmentation method for CBCT images based on Mamba and multimodal fusion. Background Technology

[0002] Oral health is an important component of overall health, and cone-beam computed tomography (CBCT), as a key imaging technique in current dental treatment, can provide complete three-dimensional structural information of teeth and alveolar bone. Accurate segmentation of tooth and alveolar bone structures from CBCT images is fundamental for constructing three-dimensional digital models and designing digital treatment plans, and is crucial for the precise implementation of treatments such as prosthodontics, orthodontics, and implantology. However, due to the extremely similar grayscale distribution and blurred boundaries between teeth and alveolar bone in CBCT images, and the frequent occurrence of severe tissue adhesions between the proximal surfaces of the crowns and the occlusal surfaces of the maxilla and mandible, achieving fully automated and accurate segmentation of tooth structures from these images remains a highly challenging task.

[0003] Currently, traditional image segmentation methods rely on manually designed features and complex artificial prior rules, resulting in cumbersome processing procedures. They also exhibit poor adaptability to automated segmentation of low-contrast, morphologically variable regions such as the alveolar bone and tooth roots, leading to limited segmentation accuracy. While existing deep learning-based 3D segmentation methods have improved segmentation accuracy to some extent, they generally fail to effectively integrate the inherent spatial coordinate geometric information of the data with textual semantic priors such as patient complaints and clinical diagnoses. This results in the model being prone to class confusion between different tooth positions when processing structures like teeth, which exhibit large intra-class differences but small inter-class differences, particularly in distinguishing symmetrical tooth positions from functional teeth with highly similar morphologies. Furthermore, mainstream 3D segmentation networks often employ self-attention mechanisms or large-scale 3D convolutions to obtain the global receptive field, leading to high computational burdens and inference latency. This makes it difficult to efficiently capture the ultra-long-range spatial dependencies of teeth along the arch curvature, failing to meet the demands of efficient, real-time response in clinical diagnosis and treatment.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a tooth segmentation method for CBCT images based on Mamba and multimodal fusion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A tooth segmentation method for CBCT images based on Mamba and multimodal fusion, comprising the following steps: Step 1: Construct a training set containing sample CBCT images and their paired case text data, preprocess the sample CBCT images, and construct a multi-channel input tensor; Step 2: Construct a tooth segmentation model, which undergoes the following training process based on the training set: Multi-scale visual feature extraction is performed on the multi-channel input tensor to obtain image bottleneck features; semantic feature extraction is performed on the case text data to obtain text semantic features; the text semantic features are dimensionally transformed by a feature adapter to align with the channel dimensions of the image bottleneck features to obtain text conditional features. Image bottleneck features and text conditional features are fused to obtain multimodal fusion features; and multi-directional sequence dependency modeling and omnidirectional aggregation are performed on the multimodal fusion features to obtain global context enhancement features. The spatial resolution of the global context enhancement features is gradually restored by upsampling until it is the same as the spatial size of the multi-channel input tensor, thus obtaining a full-size feature map; based on this full-size feature map, a prediction probability map of each voxel belonging to each tooth category is generated. The loss function is calculated based on the predicted probability map and the true label of the sample CBCT image. The parameters of the tooth segmentation model are updated through gradient backpropagation until the loss function value no longer decreases in several consecutive iterations, and the trained tooth segmentation model is obtained. Step 3: Obtain the multi-channel input tensor of the CBCT image to be segmented, and input it into the trained tooth segmentation model to obtain the tooth segmentation result of the CBCT image to be segmented.

[0007] Furthermore, constructing the multi-channel input tensor includes: The jawbone region of the sample CBCT image is cropped to obtain a three-dimensional jawbone image; the X, Y, and Z coordinate values ​​of each voxel in the normalized space are calculated with the geometric center of the three-dimensional jawbone image as the origin, and a spatial coordinate map is generated; the voxel refers to the smallest volume unit that constitutes the CBCT image. Based on the distance between each voxel position and the origin in the spatial coordinate map, and the mapping relationship that the greater the distance, the weaker the enhancement intensity, the adaptive enhancement coefficient corresponding to each voxel is calculated; the adaptive enhancement coefficient is used to perform local contrast enhancement on the corresponding voxels in the three-dimensional jawbone image to obtain the enhanced jawbone image. The enhanced jawbone image and the spatial coordinate map are stitched together in the channel dimension to form a multi-channel input tensor.

[0008] Furthermore, in the training set, each sample CBCT image corresponds to a real label; the real label is a three-dimensional annotation map with the same spatial size as the sample CBCT image, and the value of each voxel in the three-dimensional annotation map is a preset tooth category number, which includes the numbers corresponding to various types of permanent teeth, alveolar bone and background areas.

[0009] Furthermore, the multi-scale visual feature extraction is performed by the constructed three-dimensional convolutional encoder, which is composed of multiple convolutional modules and downsampling modules stacked together. It is used to extract features from the multi-channel input tensor layer by layer and reduce the spatial resolution. The feature map output from the deepest layer at the end of the three-dimensional convolutional encoder is taken as the image bottleneck feature. Semantic feature extraction of case text data is performed by a pre-trained text encoder, which is the text encoding part of a pre-trained language-visual multimodal model in the medical field. It is used to encode case text data into semantic vectors as text semantic features. The text semantic feature is a two-dimensional tensor, whose first dimension represents the sequence length dimension of the number of lexical units in the case text data, and the second dimension is the feature channel dimension.

[0010] Furthermore, the dimensional transformation is performed through a constructed feature adapter, specifically: the feature adapter first performs mean pooling on the text semantic features in the sequence length dimension to obtain aggregated text features; then, it performs channel mapping on the aggregated text features through a fully connected layer so that the number of channels is equal to the number of channels of the image bottleneck features; finally, it expands the one-dimensional feature vector obtained after mapping in the three-dimensional spatial dimensions of height, width and depth to obtain text conditional features that are completely consistent with the spatial size and number of channels of the image bottleneck features.

[0011] Furthermore, the step of fusing image bottleneck features with text conditional features is performed by the constructed spatial attention fusion module, specifically as follows: Using the spatial coordinate map generated from the sample CBCT images, a spatial attention weight map is generated through a weight mapping network consisting of convolutional modules and activation functions; The spatial attention weight map is multiplied with the text conditional features voxel by voxel to obtain the weighted text conditional features; The weighted text conditional features and the image bottleneck features are added together on a voxel basis to obtain the multimodal fusion features.

[0012] Furthermore, the steps of multi-directional sequence dependency modeling and omnidirectional aggregation are executed through the constructed Mamba scanning module and omnidirectional aggregation module, specifically as follows: The Mamba scanning module unfolds the multimodal fusion features into one-dimensional sequences according to a preset three-dimensional traversal direction, and adds positional encoding to each sequence; For each traversal direction, a forward state space model and a reverse state space model are configured. The forward sequence is input into the forward state space model and the reverse sequence is input into the reverse state space model. Sequence dependency modeling is performed on each model to obtain the bidirectional modeling output features for that traversal direction. Both the forward state space model and the reverse state space model are sequence modeling networks constructed based on discrete state space equations, and they share network parameters. The omnidirectional aggregation module calculates a weight coefficient for the output features of each traversal direction using a convolutional layer with a kernel size of 1×1×1 and an activation function; it determines the traversal direction corresponding to the largest weight coefficient as the main direction, and performs a weighted summation of the output features of the other traversal directions to obtain supplementary aggregation features; then it fuses the main direction features with the supplementary aggregation features to obtain global context enhancement features.

[0013] Furthermore, the steps of upsampling recovery and predictive probability map generation are performed by the constructed 3D convolutional decoder and the terminal classification layer. A 3D convolutional decoder is constructed in conjunction with the 3D convolutional encoder. The 3D convolutional decoder is composed of multiple upsampling modules and convolutional modules stacked together to progressively recover the spatial resolution of the global context enhancement features and finally output a full-size feature map with the same size as the multi-channel input tensor space. The full-size feature map is input into the terminal classification layer, which is a convolutional layer with the number of output channels equal to the total number of preset tooth categories. Its output is processed by a normalized exponential function to generate a predicted probability map of each voxel belonging to each tooth category. A joint loss function, which is a weighted combination of a multi-class Dessian loss function and a cross-entropy loss function, is used to calculate the error between the predicted probability map and the true label, and gradient backpropagation is performed based on this error to update the model parameters; wherein, the multi-class Dessian loss function is a pre-set weight coefficient for each tooth category.

[0014] Furthermore, prediction is performed on the CBCT image to be segmented, specifically as follows: Obtain the CBCT image to be segmented and its paired case text data; perform the same preprocessing as in step 1 on the CBCT image to be segmented to obtain the multi-channel input tensor to be segmented; the preprocessing includes jaw region cropping, spatial coordinate map generation, local contrast enhancement and channel dimension stitching; The multi-channel input tensor to be segmented and the case text data are input into the trained tooth segmentation model to obtain the predicted probability map of the CBCT image to be segmented. The tooth category corresponding to the maximum value in the channel dimension of the predicted probability map is used as the segmentation label for each voxel, and the tooth segmentation result is output.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention constructs a multi-channel input tensor that fuses and enhances jawbone images and spatial coordinate maps, providing the network with explicit prior knowledge of the absolute voxel positions. Based on a spatially adaptive local contrast enhancement strategy, it effectively strengthens the structural boundaries of low-contrast regions such as the junction of tooth roots and alveolar bone, mitigating tissue confusion caused by similar grayscale levels from the data source. Secondly, by introducing a medical pre-trained text encoder and a spatially weighted adaptive multimodal fusion mechanism, diagnostic semantic priors from case texts are dynamically injected into image features. This allows the model to obtain additional discriminative clues based on clinical descriptions when dealing with morphologically similar symmetrical teeth, suppressing inter-class confusion at the semantic level and significantly improving the accuracy of tooth position recognition. Furthermore, this invention utilizes the Mamba state space model to perform multi-directional sequence dependency modeling and omnidirectional aggregation of multimodal fusion features. With its near-linear computational complexity, it efficiently captures the ultra-long-range three-dimensional spatial dependencies of teeth along the arch curvature, significantly reducing inference latency and memory usage while maintaining global context modeling capabilities. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 This is a schematic diagram comparing the tooth segmentation results of different methods of the present invention on test samples. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] Example: Please see Figures 1 to 2 The present invention provides a technical solution: A tooth segmentation method for CBCT images based on Mamba and multimodal fusion, comprising the following steps: Step 1: Construct a training set containing sample CBCT images and their paired case text data, preprocess the sample CBCT images, and construct a multi-channel input tensor.

[0020] First, the acquired CBCT images are screened to remove irrelevant or substandard image data. The criteria for substandard quality include, but are not limited to: severe metal artifacts, motion artifacts, signal-to-noise ratio below a preset threshold (e.g., peak signal-to-noise ratio PSNR < 15 dB), and failure of the scanned field of view to completely cover the mandible or maxilla. Screening can be performed by experienced professionals or automatically using existing image quality assessment neural networks.

[0021] The selected images undergo denoising and grayscale normalization, and an adaptive contrast enhancement algorithm is used to improve the resolvability of teeth and surrounding tissue structures. The denoising process preferably employs nonlocal mean filtering or a block-matching-based 3D filtering algorithm to smooth Gaussian and quantum noise in the images while preserving details of enamel and dentin margins. Grayscale intensity normalization linearly scales the CT values ​​(Henry's units, HU) of all voxels to the [0,1] or [-1,1] range to eliminate intensity differences between different scanning devices or protocols and improve model generalization ability. The adaptive contrast enhancement algorithm specifically employs contrast-limited adaptive histogram equalization, which restricts the contrast amplification within the local neighborhood, effectively avoiding excessive noise enhancement and significantly improving the contrast of fine structures such as tooth roots and periodontal ligament spaces.

[0022] Region of Interest (ROI) extraction is performed on the enhanced image to extract the jawbone region. Specifically, this cropping operation can use threshold-based preliminary segmentation or a deep learning-based object detection network to locate the bounding box of the jawbone region. For example, bone tissue regions are initially screened based on grayscale thresholds, and then their centroids and boundaries are determined through morphological operations. Finally, the region is cropped by expanding outwards by a predetermined margin. The grayscale threshold for the bone tissue region can be determined in the following ways: In CBCT images, the CT values ​​of bone tissue are typically in the range of +300 to +1200 Henle units (HU), and an initial fixed threshold can be set accordingly to mask high-density structures; an adaptive method can also be used, such as analyzing the overall grayscale histogram of the image and selecting the intensity value corresponding to the half-width at half-maximum (FWHM) of the peak representing bone tissue as a dynamic threshold. Subsequently, image opening operations can be performed on the mask to eliminate isolated noise points, and then the largest connected component can be extracted through three-dimensional connected component analysis, as a conventional implementation of the morphological operation.

[0023] The dimensions of cropped images should be adjusted uniformly, for example, to a fixed resolution. ,like Voxels, or a consistent spatial scale based on the model structure, are used to accommodate batch input processing in neural networks. Here, H represents the number of voxels in the 3D jawbone image along the coronal axis (i.e., the height / vertical direction), also known as the height dimension; W represents the number of voxels in the 3D jawbone image along the sagittal axis (i.e., the width / anteroposterior direction), also known as the width dimension; and D represents the number of voxels in the 3D jawbone image along the transverse axis (i.e., the depth / left-right direction), also known as the depth dimension. These three parameters collectively define the spatial scale of the input 3D image.

[0024] To improve segmentation accuracy, this embodiment will construct a multi-channel input tensor. The specific process includes mandibular region cropping of the sample CBCT images, spatial coordinate map generation, spatial adaptive contrast enhancement, and channel dimension stitching, as detailed below: Jawbone region cropping: This process performs more refined jawbone region cropping on the enhanced image from the previous step, resulting in a three-dimensional jawbone image. The three-dimensional jawbone image is a [size missing]. The three-dimensional data block is the basic spatial range for all subsequent image processing and network input.

[0025] Spatial coordinate map generation: Using the geometric center of the three-dimensional jawbone image as the origin, calculate the X, Y, and Z coordinate values ​​of each voxel in the normalized space to generate a spatial coordinate map containing three channels. The specific normalization operation involves dividing the actual coordinates (x, y, z) of each voxel by half the dimensions (H, W, D) of each dimension of the image, thus scaling the range of the normalized coordinate values ​​to... The interval is calculated using the following formula: Among them, a voxel refers to the smallest volume unit that constitutes a CBCT image. For any voxel in a three-dimensional jawbone image, its spatial position can be determined by displacement using three integer coordinates (x, y, z), where x represents the index position in the height dimension, y represents the index position in the width dimension, and z represents the index position in the depth dimension. This represents the normalized relative coordinate value in the height dimension. This represents the normalized relative coordinate value along the width dimension. This represents the normalized relative coordinate value along the depth dimension. These are the coordinates of the three-dimensional geometric center of the jawbone image. These represent the index positions in the height, width, and depth dimensions, respectively.

[0026] This normalization method encodes the relative spatial position information of each voxel with respect to the center of the jawbone, providing the network with a unified spatial reference system. This helps the model understand the global anatomical structure of teeth, such as symmetry and relative positional relationships. For voxels in the image edge region, the absolute value of their normalized coordinates approaches 1; for voxels in the central region, the normalized coordinate value approaches 0.

[0027] Spatial adaptive contrast enhancement: Based on the distance of each voxel position in the spatial coordinate map to the origin, and following a mapping relationship where the greater the distance, the weaker the enhancement intensity, an adaptive enhancement coefficient is calculated for each voxel. This adaptive enhancement coefficient is then used to perform local contrast enhancement on the corresponding voxels in the 3D jawbone image, resulting in an enhanced jawbone image.

[0028] This step is designed based on a priori medical knowledge: in clinical diagnosis, physicians typically focus their attention on the central region of the image field of view, i.e., the core area of ​​the dental arch; while peripheral regions are mostly background or the periphery of anatomical structures, and excessive enhancement may introduce unnecessary noise or artifacts. Therefore, an adaptive enhancement strategy based on spatial distance is adopted to enhance the image selectively and purposefully. Specifically, the adaptive enhancement coefficient can be calculated using a spatial weighting function centered at the origin, such as a Gaussian function or its variants. A preferred implementation is as follows: First, calculate the Euclidean distance r of each voxel from the origin, using the following formula: Then, the enhancement factor is calculated based on this distance. The calculation formula is as follows: ;in, For maximum additional strength, The reinforcement intensity is used as a baseline, reflecting the difference in reinforcement requirements between the central and peripheral regions. To control the decay rate; and All of these are hyperparameters.

[0029] Regarding the determination of parameters: The possible value is 0.5. The possible value is 1.0. This value means that the enhancement weight of voxels in the image center is 1.5 times; while the enhancement weight of voxels in the edge regions is increased by a factor of 1.0. As the gain increases, the gain coefficient smoothly decreases from 1.5 to nearly 1.0, maintaining the original boost level. Parameter To control the decay rate, a value of 3.0 can be set, ensuring that the normalized distance... When it is 0.5, the additional strength is increased. Decay to approximately This achieves a significant enhancement in the central region and a smooth transition to the edges. Finally, the calculated adaptive enhancement coefficients are... Multiplying by each voxel corresponding to the 3D jawbone image performs local, adaptive contrast enhancement, generating an enhanced jawbone image. This operation is a voxel-by-voxel multiplication operation, i.e. ,in, This represents the grayscale intensity value at the same spatial location (x, y, z) after spatial adaptive contrast enhancement processing. This is the cropped image of the jawbone. This represents the original grayscale intensity value (or voxel value) of the jawbone image after cropping and resizing at spatial location (x,y,z).

[0030] Multi-channel input tensor construction: The enhanced jawbone image and the spatial coordinate map are concatenated along the channel dimension to form a multi-channel input tensor. After concatenation, the number of channels in the input tensor changes from 1 (original grayscale image) to 4 (1 enhanced grayscale channel + 3 coordinate channels). This multi-channel input tensor serves as the input to the 3D convolutional encoder, containing both spatially adaptively enhanced intensity information and explicit spatial location prior information, providing a richer input representation for subsequent feature learning.

[0031] The paired case text data is cleaned to remove noisy characters and non-critical descriptions, and then segmented, labeled, and embedded in the text for encoding to adapt to subsequent feature extraction and multimodal fusion tasks. The data cleaning steps include removing special characters, redundant spaces, and colloquial descriptions unrelated to dental anatomy. Segmentation aims to divide the continuous text sequence into discrete words, using algorithms such as word slices or byte pairs. The labeling and encoding process utilizes the vocabulary of a pre-trained text encoder to map each word to a unique integer ID, forming a word index sequence, which serves as the input format for the subsequent BiomedCLIP pre-trained model. Through this preprocessing, unstructured free text is transformed into a computer-processable, standardized sequence of numerical vectors.

[0032] Step 2: Construct a tooth segmentation model, which undergoes the following training process based on the training set: Multi-scale visual feature extraction is performed on the multi-channel input tensor to obtain image bottleneck features; semantic feature extraction is performed on the case text data to obtain text semantic features; the text semantic features are dimensionally transformed by a feature adapter to align with the channel dimensions of the image bottleneck features to obtain text conditional features.

[0033] In this embodiment, each sample CBCT image in the training set corresponds to a ground truth label. The ground truth label is a 3D annotation map with the same spatial dimensions as the sample CBCT image. The value of each voxel in this 3D annotation map is a preset tooth category number. The preset tooth category number includes the numbers corresponding to various types of permanent teeth (such as central incisors, lateral incisors, canines, first premolars, second premolars, first molars, second molars, and third molars), alveolar bone, and background regions. For example, in a standard dental segmentation task containing 32 permanent teeth, the total number of categories C can be set to 34 (32 categories of teeth + 1 category of alveolar bone + 1 category of background). This fine-grained category definition is the basis for improving fine-grained recognition capabilities in this embodiment.

[0034] The multi-scale visual feature extraction is performed by a constructed 3D convolutional encoder. In this embodiment, the specific structure of the 3D convolutional encoder is as follows: it contains 5 convolutional modules and 2 downsampling modules. Each convolutional module consists of a 3D convolutional layer with a kernel size of 3×3×3 and an instance normalization layer, and uses LeakyReLU (with a negative slope of 0.01) as the non-linear activation function. The downsampling module is implemented by a 3D convolutional layer with a kernel size of 3×3×3 and a stride of 2, used to halve the spatial resolution of the feature map while doubling the number of channels. The input multi-channel input tensor (size 1×H×W×D, where 1 is the number of channels after merging the original grayscale channel with the three spatial coordinate channels) passes through 3 consecutive convolutional modules and 1 downsampling module, then through 2 more convolutional modules and 1 more downsampling module, and finally outputs a size of [missing information] at the end of the encoder. The feature map is taken as the image bottleneck feature; where This represents the number of channels for the image bottleneck feature.

[0035] Semantic feature extraction from the case text data is performed by a pre-trained text encoder. In this embodiment, the pre-trained text encoder is the text encoder part of the BiomedCLIP language-visual multimodal model pre-trained in the medical field. This model is pre-trained on massive biomedical image-text pairs and can map the input text description to a unified embedding space rich in semantic information. For a case text prompt formed after preprocessing and structuring, such as a patient's missing left upper first molar and mesial impaction of the right lower third molar, the text encoder converts it into a two-dimensional tensor form of text semantic features, with a size of [missing information]. Where L represents the length of the word sequence after text segmentation. The feature dimensions of the BiomedCLIP text encoder (e.g., 768 dimensions).

[0036] In this embodiment, the dimensionality transformation is performed through a constructed feature adapter. Specifically, the feature adapter first performs mean pooling on the text semantic features along the sequence length dimension L to obtain an aggregated text feature vector with a size of [missing information]. Subsequently, a fully connected layer is used to perform channel mapping on the aggregated text features, so that the number of channels is equal to the number of channels of the image bottleneck features. Finally, the one-dimensional feature vector obtained after mapping (with a size of...) The reshaping operation transforms the result into [1,1,1,] The four-dimensional tensor of ], and then at height ( ),width( ) and depth ( Broadcasting is performed in three spatial dimensions to generate images with bottleneck feature spatial dimensions ( ) and number of channels ( The text features are completely consistent. This process ensures that the semantic information of the text is evenly tiled in three-dimensional space, providing a foundation for subsequent spatially adaptive fusion.

[0037] Image bottleneck features and text conditional features are fused to obtain multimodal fusion features; and multi-directional sequence dependency modeling and omnidirectional aggregation are performed on the multimodal fusion features to obtain global context enhancement features.

[0038] In this embodiment, the step of fusing image bottleneck features with text conditional features is performed by the constructed spatial attention fusion module, specifically: using a spatial coordinate map (size: ...) generated from the sample CBCT image. A spatial attention weight map is generated through a weight mapping network consisting of two cascaded convolutional modules (each with the same structure as described above) and a sigmoid activation function. (size is) The value range of this spatial attention weight map is (0,1). The physical meaning of this spatial attention weight map is that the model can adaptively learn how to give higher response weights to text features at locations that require more prior knowledge, such as the center or boundary of the teeth, while suppressing their influence in uniform background areas.

[0039] The spatial attention weight map With the text conditional features Multiply each voxel to obtain the weighted text conditional features; then combine the weighted text conditional features with the image bottleneck features. Multimodal fusion features are obtained by adding voxels together. The calculation formula is as follows: in, This represents the Hadamard Product, which is element-wise multiplication. This fusion method allows textual semantic priors to be dynamically injected into visual features based on the importance of spatial location.

[0040] In this embodiment, the steps of multi-directional sequence dependency modeling and omnidirectional aggregation are executed by the constructed Mamba scanning module and omnidirectional aggregation module. Specifically, the Mamba scanning module, according to a preset three-dimensional traversal direction, models the multi-directional sequence dependency modeling and omnidirectional aggregation module with a size of [missing information]. Multimodal fusion features Each sequence is expanded into a one-dimensional sequence, and a positional code is added to each sequence.

[0041] The preset three-dimensional traversal directions include six types: height-width-depth (HWD), height-depth-width (HDW), width-height-depth (WHD), width-depth-height (WDH), depth-height-width (DHW), and depth-width-height (DWH).

[0042] For each configuration, the 3D voxel data is read sequentially according to the traversal path, with a flattened length of [missing information]. A one-dimensional sequence. To compensate for the loss of spatial location information caused by flattening, the following sine and cosine position codes P are added to each position pos in the sequence: in, pos represents the position index in the flattened one-dimensional sequence. For a sequence of length L unfolded in a certain traversal direction, pos takes the value 0, 1, 2, ..., L-1. It provides a unique sequence number for each voxel in its current serialization direction. This represents the value of the 2i-th dimension (i.e., the even-numbered dimension) of the position encoding vector at position pos. This represents the value of the (2i+1)th dimension (i.e., the odd-numbered dimension) of the position encoding vector at position pos. i is the dimension index of the feature channel, ranging from [0, ..., ... / 2).

[0043] The six three-dimensional traversal directions described above are specifically designed based on the unique spatial structure characteristics of three-dimensional dental CBCT image data and the directional sensitivity of the Mamba state space model when processing sequence data.

[0044] Specifically, a single-direction sequence unfolding method can only capture spatial dependencies along a specific axis. Furthermore, teeth are arranged in a horseshoe-shaped, non-rigid configuration within the jawbone along the dental arch curve. Their long-range three-dimensional spatial dependencies are not distributed along a single coordinate axis, but are simultaneously reflected in the coronal, sagittal, and transverse orthogonal directions and the intersecting paths they form. For example, sequence unfolding along the height direction (H) helps model the hierarchical structure from the crown to the root apex; sequence unfolding along the width direction (W) is beneficial for capturing the adjacent relationships between the buccal and lingual teeth and the alveolar bone; and sequence unfolding along the depth direction (D) effectively correlates the spatial order between adjacent teeth distributed along the curved path of the dental arch.

[0045] The combination of six traversal directions essentially involves exhaustively decomposing the 3D voxel space along various orthogonal directions, constructing a multi-view representation set covering all principal axis traversal sequences. Using six traversal directions eliminates serialization direction bias, enabling isotropic long-range dependency modeling. Mamba, as a unidirectional sequence model, inherently depends on the direction of its sequence reading. Using only a single traversal direction (such as HWD) can lead to an overemphasis on dependencies in a specific spatial direction, weakening or even ignoring contextual information in other directions. Configuring all six permutations of traversal directions is equivalent to serializing and perceiving the 3D feature map from all possible orthogonal perspectives, ensuring that spatial dependencies in any direction are not missed or weakened, thus achieving global context capture. Furthermore, this traversal method provides rich alternative representations for the adaptive optimization of the omnidirectional aggregation module. Significant individual differences exist in the oral anatomy of different patients, and the dominant spatial dependency directions also differ at different tooth positions within the same patient. For example, in the incisor region, the adjacency relationship in the width direction may be the most critical; in the molar region, the path dependency along the curved path of the posterior segment of the dental arch in the depth direction may dominate. The six traversal directions provide a sufficient candidate direction pool for the subsequent omnidirectional aggregation module, enabling the model to dynamically select the direction with the richest information and most relevant to the current segmentation task as the main direction based on the specific anatomical position of the current voxel, guided by spatial coordinate priors, through a weight learning mechanism. The information of the remaining directions is used as supplementary information, thereby achieving spatially adaptive sequence dependency aggregation.

[0046] For each traversal direction, a forward state space model (Forward SSM) and a backward state space model (Backward SSM) are configured. The forward sequence is input into the forward state space model, and the reverse sequence (i.e., the forward sequence is reversed) is input into the backward state space model. Sequence dependency modeling is performed on each model to obtain the bidirectional modeling output features for that traversal direction. Both the forward and backward state space models are sequence modeling networks constructed based on discrete state space equations. The forward and backward state space models are two network module instances with identical structures, sharing all network parameters during training, i.e., sharing the same set of state matrices, control matrices, output matrices, and discretization step sizes.

[0047] The internal network construction and parameter settings of the bidirectional state space modeling module are as follows: From input to output, this unit first includes a linear projection layer to receive the input vector of the current time step. This input vector is the result of fusing the processed hidden state of the previous time step with the current input, and its dimension is equal to the number of feature channels output by the encoder. This linear projection layer is an unbiased fully connected layer, and its weight matrix is ​​the control matrix. The size of the control matrix is ​​the hidden state feature dimension multiplied by the input feature dimension, where the hidden state feature dimension is a hyperparameter preset during model design. In this embodiment, it is set to 16 to achieve a balance between model expressiveness and computational efficiency. After linear projection, a projection vector is obtained.

[0048] This unit contains a core state-space computation layer, which maintains a hidden state with the same dimension as the hidden state feature dimension mentioned above. This core computation layer operates according to the discrete state-space equations. Specifically, it first uses a fully connected layer with an activation function to linearly transform the hidden state of the previous time step. The weight matrix of this fully connected layer is the state matrix, with a size equal to the product of the hidden state feature dimensions, representing the transition relationships within the hidden state itself. The activation function used can be, for example, the SiLU function. Then, the transformed hidden state of the previous time step is added element-wise to the projection vector output by the aforementioned linear projection layer to obtain the updated hidden state of the current time step. Afterward, the core computation layer uses another unbiased fully connected layer to linearly transform the hidden state of the current time step, obtaining an intermediate output vector. The weight matrix of this fully connected layer is the output matrix, with a size equal to the product of the input feature dimensions and the hidden state feature dimensions, representing the mapping relationship from the hidden state to the output.

[0049] This unit also includes an output linear transformation layer that receives the intermediate output vector from the core state space computation layer. The vector is then scaled and offset by a biased fully connected layer to obtain the final output vector for that time step, whose dimension is restored to the same number of original input feature channels.

[0050] Regarding parameter initialization and settings, to achieve discretization, a trainable scalar parameter is defined within the core state space computation layer. This parameter, after being mapped by a flexible additive function, yields the discretization step size. The initial value of this scalar parameter can be set to a small positive value (e.g., 0.001) to enable the model to capture long-range dependencies in the early stages of training. All trainable matrices, including the state matrix, control matrix, and output matrix, can be initialized using the Xavier uniform initialization method, while the bias term is initialized to zero. When constructing bidirectional sequence dependency modeling, the specific approach is as follows: A forward state space model instance and a backward state space model instance are constructed, both with identical network structures and forcibly sharing all the aforementioned weight matrices and trainable step size parameters. The forward sequence is input into the forward instance to obtain the forward feature sequence; simultaneously, the backward sequence obtained by flipping the original sequence is input into the backward instance to obtain the backward feature sequence. However, this output needs to be flipped again to restore the alignment with the corresponding positions in the forward sequence, ultimately resulting in a backward feature sequence aligned with the positions of the forward feature sequence.

[0051] The modeling process of the state-space model follows the discrete-time state-space equations, and its calculation at each time step t is as follows: in, This represents the hidden state at time t. This represents the hidden state at the previous time step t-1. Let be the input features at time t. Let be the output at time t, where t represents the time variable; state matrix A, control matrix B, and output matrix C are all trainable parameter matrices. Specifically, for the initial time of the sequence, i.e., t=0, the initial hidden state of the forward state space model is initialized as a zero vector with the same dimension as the hidden state feature dimension; similarly, the initial hidden state of the inverse state space model is also initialized as a zero vector with the same dimension as the zero vector of the initial hidden state of the forward state space model. and These are the parameters after discretization using the zero-order preservation rule, and their calculation method is as follows: in, It is an identity matrix with the same dimensions as the state matrix A, that is, a square matrix with all elements on the main diagonal being 1 and all other elements being 0. The time step is used to discretize the continuous-time system. In this embodiment, it is also set as a trainable parameter with an initial value of 0.001, so that the model can capture long-range dependencies in the early stage of training and make adaptive adjustments according to the global optimization objective during the training process.

[0052] Finally, the features output from the ascending sequence Features of the reverse sequence output By concatenating the features along the feature dimension, we obtain the bidirectional modeling output features in that direction.

[0053] The omnidirectional aggregation module calculates a weight coefficient for the output features in each traversal direction using a 1×1×1 convolutional layer and a sigmoid activation function. The traversal direction corresponding to the maximum weight coefficient is determined as the main direction. The output features of the other traversal directions are weighted and summed to obtain supplementary aggregated features. The main direction features are then fused with the supplementary aggregated features to obtain global context enhancement features. The fusion calculation formula is as follows: in, This represents the weighted sum of features across all traversal directions. Its function is to output the model for all directions based on the learned weights. Weighted aggregation is performed to form a comprehensive feature containing information from multiple perspectives. N=6 represents the total number of traversal directions, and d represents the direction index. To model and output features for the d-th direction, is the weight coefficient for the d-th direction. Indicates residual supplementation characteristics, Maximum weight The corresponding main directional features. This represents a 1×1×1 transposed convolution operation used to aggregate supplementary features. Perform a linear transformation to align with the main direction features. Better alignment and fusion in feature space.

[0054] The core idea of ​​this aggregation mechanism is that the main direction with the largest weight contributes the most significant structural information, while the weighted sum of the other directions provides supplementary, multi-faceted detailed information. The combination of the two can produce an output that is comprehensive in information and has strong feature representation capabilities.

[0055] The spatial resolution of the global context enhancement features is gradually restored by upsampling until it is the same as the spatial size of the multi-channel input tensor, thus obtaining a full-size feature map; based on this full-size feature map, a prediction probability map of each voxel belonging to each tooth category is generated.

[0056] In this embodiment, the steps of upsampling restoration and predictive probability map generation are performed by the constructed 3D convolutional decoder and the terminal classification layer. A 3D convolutional decoder is constructed to cooperate with the 3D convolutional encoder. This decoder consists of two upsampling modules and three convolutional modules stacked together, specifically in the following stacking order: first upsampling module, first convolutional module, second upsampling module, second convolutional module, and third convolutional module. Each upsampling module consists of a trilinear interpolation layer and a 3×3×3 transposed convolutional layer with a stride of 1, used to progressively double the resolution of the feature map while progressively halving the number of channels, ultimately outputting a full-size feature map with the same size (H×W×D) as the multi-channel input tensor space, and its number of channels is [missing information]. .

[0057] The full-size feature map is input into the terminal classification layer, which is a convolutional layer with a kernel size of 1×1×1 and an output channel number equal to the preset total number of tooth categories C. Its output is processed by a normalized exponential function (Softmax) to generate a predicted probability map of each voxel belonging to each tooth category. The probability formula for a given voxel belonging to the c-th class is: in, Indicates known input Under the given conditions, the probability that the voxel belongs to class c is... Let c be the output class label, indicating that the voxel is predicted to be of class c, and X be the input to the model, i.e., the feature corresponding to the voxel. This is the output logical value of the voxel corresponding to the r-th category in the classification layer. c and r both represent predefined category indices in the tooth segmentation task, and C represents the total number of predefined categories in the tooth segmentation task. Finally, the category with the highest probability value is selected as the segmentation label for that voxel.

[0058] The loss function is calculated based on the predicted probability map and the true label of the sample CBCT image. The parameters of the tooth segmentation model are updated through gradient backpropagation until the loss function value no longer decreases in several consecutive iterations, thus obtaining the trained tooth segmentation model.

[0059] If the loss function value no longer decreases after several consecutive iterations, the model is considered to have converged. The convergence condition is determined as follows: A monitoring metric is set, such as the joint loss function value calculated on the test set. A preset tolerance number of training epochs is also set; in this embodiment, this number is set to 50 iterations. After each training epoch, the joint loss function value of the current model on the validation set is calculated, and the best-ever joint loss function value is recorded. If the current joint loss function value is lower than the best-ever joint loss function value, the best-ever joint loss function value is updated, the current model parameters are saved, and the current epoch count is reset to 0. If the current joint loss function value is not lower than the best-ever joint loss function value, the current epoch count is incremented by 1. When the current epoch count reaches the preset tolerance number of epochs (i.e., 50 iterations), it is determined that the model performance no longer shows significant improvement, and the training process terminates. Finally, the saved model parameters corresponding to the best-ever joint loss function value are selected as the parameters of the trained tooth segmentation model.

[0060] A joint loss function, composed of a weighted combination of a multi-class Descein loss function and a cross-entropy loss function, is used to calculate the error between the predicted probability map and the true label. Gradient backpropagation is then performed based on this error to update the model parameters. The multi-class Descein loss function has pre-set weight coefficients for each tooth category. The specific expression of the joint loss function L is as follows: in, Let cross-entropy be the loss function. To balance the contributions of the two loss terms, a hyperparameter is used. In this embodiment, The value of is set to 1. This setting ensures that the two loss terms have similar magnitudes in the initial stage, allowing the model training to balance overall classification accuracy and recall for small object categories. The specific value can be determined through parameter search experiments on a mini-batch validation set.

[0061] Multiclass Dess loss function The calculation formula is: Where J is the total number of elements in a single volume data, and a single volume data refers to a complete 3D CBCT image sample used for a single input model training. Let represent the probability that the j-th voxel is predicted to be of class c. This indicates whether the j-th voxel belongs to category c in the true label (1 for belonging, 0 for not belonging), where j represents the voxel index. It is the weight coefficient of category c, used to alleviate the problem of severe imbalance in the number of samples of each category in the dataset (such as background voxels being far more numerous than specific tooth voxels). It is a very small smoothing constant, let it be... This is used to prevent calculation errors caused by a denominator of zero.

[0062] In this embodiment, It is calculated based on the reciprocal of the frequency of each category in the entire training set, that is... ,in, This represents the frequency of category c across all sample voxels. This inverse frequency-weighted logarithmic smoothing method significantly increases the contribution of small, infrequent tooth categories like canines or third molars to the loss function, while avoiding excessive weighting for rare, noisy label categories. It's a reasonable choice that balances effectiveness and training stability. This weight can be pre-calculated using global statistics on the entire training set before training begins and remains constant during training. 1.02 is a minimal offset constant introduced to prevent zero or negative values ​​in logarithmic operations; 1.02 is chosen to ensure that even... The argument of the logarithm +1.02 ≥ 1.02, and its natural logarithm is greater than zero, thus ensuring that the denominator is strictly greater than zero and the weight values ​​are bounded. This number, slightly greater than 1, provides a safety margin to prevent numerical instability. Its specific value (1.02) can be fine-tuned experimentally, but theoretically any value slightly greater than 1 (such as 1.01, 1.001) is feasible, and 1.02 is a reasonable empirical choice.

[0063] Step 3: Obtain the multi-channel input tensor of the CBCT image to be segmented, and input it into the trained tooth segmentation model to obtain the tooth segmentation result of the CBCT image to be segmented.

[0064] In this embodiment, the prediction of the CBCT image to be segmented is performed as follows: Obtain the CBCT image to be segmented and its paired case text data; perform the same preprocessing as in step 1 on the CBCT image to be segmented to obtain the multi-channel input tensor to be segmented; the preprocessing includes jaw region cropping, spatial coordinate map generation, local contrast enhancement, and channel dimension stitching. Specifically, the above preprocessing process adopts the same parameter settings and operation sequence as the training phase to ensure that the distribution of input data during model inference is consistent with that during training, thereby ensuring the stability of segmentation performance. The parameter settings include, but are not limited to: the target spatial resolution H×W×D after cropping (e.g., 256×256×128 voxels), the target interval for grayscale intensity normalization (e.g., [0,1]), the maximum additional enhancement intensity in spatial adaptive contrast enhancement (e.g., 0.5), the baseline enhancement intensity (e.g., 1.0), and the decay rate (e.g., 3.0). The method for determining the values ​​of these parameters is the same as described in step 1, and will not be repeated here.

[0065] The obtained paired case text data undergoes the same text preprocessing as in step 1, including: removing special characters, redundant spaces, and colloquial descriptions unrelated to dental anatomy; segmenting the cleaned text into discrete word sequences using the same word segmentation algorithm as in the training phase (e.g., word slice or byte pair encoding algorithm); and mapping each word to a unique integer ID using the vocabulary provided by the BiomedCLIP pre-trained text encoder, forming a word index sequence. This word index sequence is the preprocessed structured representation of the case text data to be segmented and can be directly input into the subsequent text encoder.

[0066] The multi-channel input tensor to be segmented and the case text data are input into the trained tooth segmentation model to obtain the predicted probability map of the CBCT image to be segmented. The trained tooth segmentation model is the model instance saved after training convergence in step 2, which has the historical best joint loss function value. All trainable parameters of this model include the 3D convolutional encoder, the BiomedCLIP pre-trained text encoder, the fully connected layer in the feature adapter, the weight mapping network in the spatial attention fusion module, and the state matrix, control matrix, output matrix, and discretization step size in the Mamba scanning module. All weights and biases of the 1×1×1 convolutional layer in the omnidirectional aggregation module, as well as the 3D convolutional decoder and the terminal classification layer, have been determined and fixed during the training phase and will not be updated during inference.

[0067] During the inference phase, the forward propagation process of the model is basically the same as the training forward propagation process described in step 2, except that the loss function calculation and gradient backpropagation are not performed.

[0068] The tooth category corresponding to the maximum value in the channel dimension of the predicted probability map is used as the segmentation label for each voxel, and the tooth segmentation result is output. Specifically, for the voxel at spatial location (x, y, z), its segmentation label is... The method for determining it is as follows: Where c∈{1,2,…,C}, and C is the total number of preset tooth categories. Indicates taking Find the category index c that yields the maximum value. Perform this operation on all H×W×D voxels to obtain a 3D segmentation label map with the same spatial dimensions as the input image. In this segmentation label map, each voxel is assigned a unique category number, corresponding to the central incisor, lateral incisor, canine, first premolar, second premolar, first molar, second molar, third molar, alveolar bone, or background region.

[0069] Qualitative and quantitative assessments can be performed on the output tooth segmentation results. Qualitative assessment involves overlaying the segmentation results in different colors onto the original CBCT image for visual review by a professional dentist to determine whether the boundaries of each tooth conform to the anatomical structure and whether there are any obvious missegmented or omitted areas. Quantitative assessment is achieved by calculating multiple evaluation metrics between the segmentation results and the true labels (if any), including but not limited to: Dessian similarity coefficients for each category, mean surface distance, and Hausdorff distance.

[0070] For example, the formula for calculating the Dessian similarity coefficient is: ,in, This represents the Dessian similarity coefficient belonging to category c. This represents the set of voxels predicted by the model to be of class c. Let represent the set of voxels belonging to category c in the true labels, |·| represent the number of voxels in the set, and ∩ represent the intersection of the two sets. The value range is [0,1]. The closer it is to 1, the higher the degree of overlap between the segmentation result and the true label in that category, and the better the segmentation accuracy.

[0071] The tooth segmentation model constructed in this embodiment adopts an encoder-decoder structure and incorporates text-conditional guidance and multi-directional sequence modeling branches. The model input consists of two parts: one is a preprocessed multi-channel input tensor containing enhanced jawbone images and spatial coordinate maps, and the other is paired case text data. The multi-channel input tensor first enters a 3D convolutional encoder, which is composed of multiple convolutional modules and downsampling modules stacked alternately. It extracts multi-scale visual features layer by layer and reduces spatial resolution, finally outputting the deepest feature map at the end of the encoder as the image bottleneck feature. At the same time, the case text data is fed into a text encoder based on a pre-trained language-visual multimodal model in the medical field, encoding it into a two-dimensional tensor form of text semantic features rich in semantic information. This text semantic feature then enters the feature adapter, and after mean pooling aggregation in the sequence length dimension, it is mapped to the same number of channels as the number of channels in the image bottleneck feature through a fully connected layer. Then, it is broadcast expanded in the three-dimensional spatial dimension to generate text conditional features that are completely aligned with the spatial size and number of channels of the image bottleneck feature. Subsequently, image bottleneck features and text conditional features are input in parallel into a spatial attention fusion module. This module utilizes a spatial coordinate map generated from the input image and, through a weight mapping network composed of convolutional modules and activation functions, produces a spatially adaptive attention weight map. This map is used to weight the text conditional features voxel-by-voxel, and the weighted result is added to the image bottleneck features voxel-by-voxel to obtain multimodal fusion features. These features are then fed into a sequence dependency modeling unit consisting of a Mamba scanning module and an omnidirectional aggregation module. In the Mamba scanning module, the multimodal fusion features are unfolded into one-dimensional sequences and positional encodings are added according to six preset three-dimensional traversal directions. Each direction is configured with forward and backward state space models sharing parameters, performing bidirectional dependency modeling on both the forward and reverse sequences, and outputting bidirectional modeling features for each traversal direction. The output features from all traversal directions are aggregated into the omnidirectional aggregation module. This module calculates weight coefficients for each direction's features using 1×1×1 convolutional layers and activation functions, using the direction with the largest weight as the main direction. The features from the remaining directions are weighted and summed as supplementary aggregation features, and the main direction features are fused with the supplementary aggregation features to obtain global context enhancement features. The feature map then enters a 3D convolutional decoder, which consists of multiple upsampling and convolutional modules symmetrically stacked to gradually restore the spatial resolution of the feature map until it outputs a full-size feature map with the same spatial size as the original multi-channel input tensor. Finally, this full-size feature map passes through a terminal classification layer consisting of 1×1×1 convolutional layers. After being processed by a normalized exponential function, it outputs a predicted probability map of each voxel belonging to each preset tooth category, completing the entire forward propagation process from input to segmentation prediction.

[0072] To objectively verify the segmentation performance of the method in this embodiment under complex clinical scenarios, 30 representative CBCT scan data were selected for quantitative evaluation. The samples covered typical challenging scenarios such as complete dentition, crowded dentition, metal artifact interference, impacted teeth, alveolar bone resorption, and motion artifacts. The method in this embodiment was compared with 3D U-Net, V-Net, nnU-Net, Swin UNETR, and the Mamba-only unimodal model containing only image branches, with the Destination similarity coefficient (DSC) used as the core evaluation index. Detailed segmentation results of the 30 test samples are shown in Table 1, along with scatter plots and trend comparisons of the DSC distribution for each method. Table 1. Example of DSC comparison results for tooth segmentation on test samples using different methods: From the data in Table 1 and Figure 2 The distribution trend clearly shows that the DSC values ​​of the method in this embodiment are significantly higher than the other five comparative methods on all 30 test samples. The data points are generally located in the uppermost region of the scatter plot, demonstrating a stable and consistent performance advantage. In samples with extremely high segmentation difficulty, such as sample 17 (impacted tooth, partially erupted tooth) and sample 24 (motion artifact, low image quality), the DSC values ​​are approximately 3.95 and 4.19 percentage points higher than the single-modal optimal baseline Mamba-only, respectively. This indicates that by introducing semantic priors from the case text, the model can effectively overcome ambiguities caused by image quality degradation or morphological abnormalities by utilizing tooth position cues from clinical descriptions. In the scenario of obvious metal artifacts in sample 05, the DSC of the method of this invention remains above 0.90, while nnU-Net and lower methods all drop below 0.85. This is due to the enhancement of the effective signal in the central region by the spatial adaptive contrast enhancement module and the ultra-long-range dependency modeling capability of the Mamba state space model along the dental arch curve, which allows the local information loss in the metal artifact region to be filled in by the global anatomical structure. Figure 2 From the trend line, the method in this embodiment has the narrowest DSC fluctuation range (0.8847-0.9421) among 30 samples, with a standard deviation of only 0.0168, which is much lower than Swin UNETR's 0.0264 and 3D U-Net's 0.0327. This indicates that the multimodal fusion strategy and omnidirectional aggregation mechanism endow the model with good cross-sample robustness, and can maintain stable high-precision segmentation output for different dental arch states and image quality.

[0073] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0074] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0076] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A tooth segmentation method for CBCT images based on Mamba and multimodal fusion, characterized in that, The specific steps include: Step 1: Construct a training set containing sample CBCT images and their paired case text data, preprocess the sample CBCT images, and construct a multi-channel input tensor; Step 2: Construct a tooth segmentation model, which undergoes the following training process based on the training set: Multi-scale visual feature extraction is performed on the multi-channel input tensor to obtain image bottleneck features; semantic feature extraction is performed on the case text data to obtain text semantic features; the text semantic features are dimensionally transformed by a feature adapter to align with the channel dimensions of the image bottleneck features to obtain text conditional features. Image bottleneck features and text conditional features are fused to obtain multimodal fusion features; and multi-directional sequence dependency modeling and omnidirectional aggregation are performed on the multimodal fusion features to obtain global context enhancement features. The spatial resolution of the global context enhancement features is gradually restored by upsampling until it is the same as the spatial size of the multi-channel input tensor, thus obtaining a full-size feature map; based on this full-size feature map, a prediction probability map of each voxel belonging to each tooth category is generated. The loss function is calculated based on the predicted probability map and the true label of the sample CBCT image. The parameters of the tooth segmentation model are updated through gradient backpropagation until the loss function value no longer decreases in several consecutive iterations, and the trained tooth segmentation model is obtained. Step 3: Obtain the multi-channel input tensor of the CBCT image to be segmented, and input it into the trained tooth segmentation model to obtain the tooth segmentation result of the CBCT image to be segmented.

2. The CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 1, characterized in that, Constructing the multi-channel input tensor includes: The jawbone region of the sample CBCT image is cropped to obtain a three-dimensional jawbone image; the X, Y, and Z coordinate values ​​of each voxel in the normalized space are calculated with the geometric center of the three-dimensional jawbone image as the origin, and a spatial coordinate map is generated; the voxel refers to the smallest volume unit that constitutes the CBCT image. Based on the distance between each voxel position and the origin in the spatial coordinate map, and the mapping relationship that the greater the distance, the weaker the enhancement intensity, the adaptive enhancement coefficient corresponding to each voxel is calculated; the adaptive enhancement coefficient is used to perform local contrast enhancement on the corresponding voxels in the three-dimensional jawbone image to obtain the enhanced jawbone image. The enhanced jawbone image and the spatial coordinate map are stitched together in the channel dimension to form a multi-channel input tensor.

3. The CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 1, characterized in that, In the training set, each sample CBCT image corresponds to a real label; the real label is a three-dimensional annotation map with the same spatial size as the sample CBCT image. The value of each voxel in the three-dimensional annotation map is a preset tooth category number, which includes the numbers corresponding to various types of permanent teeth, alveolar bone, and background areas.

4. The CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 1, characterized in that, The multi-scale visual feature extraction is performed by a constructed three-dimensional convolutional encoder, which is composed of multiple convolutional modules and downsampling modules stacked together. It is used to extract features from the multi-channel input tensor layer by layer and reduce the spatial resolution. The feature map output from the deepest layer at the end of the three-dimensional convolutional encoder is taken as the image bottleneck feature. Semantic feature extraction of case text data is performed by a pre-trained text encoder, which is the text encoding part of a pre-trained language-visual multimodal model in the medical field. It is used to encode case text data into semantic vectors as text semantic features. The text semantic feature is a two-dimensional tensor, whose first dimension represents the sequence length dimension of the number of lexical units in the case text data, and the second dimension is the feature channel dimension.

5. The CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 4, characterized in that, The dimensional transformation is performed through a constructed feature adapter, specifically: the feature adapter first performs mean pooling on the text semantic features in the sequence length dimension to obtain aggregated text features; then, a fully connected layer performs channel mapping on the aggregated text features so that the number of channels is equal to the number of channels of the image bottleneck features; Finally, the one-dimensional feature vector obtained after mapping is expanded in the three-dimensional space dimensions of height, width, and depth to obtain text conditional features that are completely consistent with the feature space size and number of channels of the image bottleneck.

6. The CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 2, characterized in that, The step of fusing image bottleneck features with text conditional features is performed by the constructed spatial attention fusion module, specifically as follows: Using the spatial coordinate map generated from the sample CBCT images, a spatial attention weight map is generated through a weight mapping network consisting of convolutional modules and activation functions; The spatial attention weight map is multiplied with the text conditional features voxel by voxel to obtain the weighted text conditional features; The weighted text conditional features and the image bottleneck features are added together on a voxel basis to obtain the multimodal fusion features.

7. The CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 1, characterized in that, The steps of multi-directional sequence dependency modeling and omnidirectional aggregation are executed through the constructed Mamba scanning module and omnidirectional aggregation module, specifically as follows: The Mamba scanning module unfolds the multimodal fusion features into one-dimensional sequences according to a preset three-dimensional traversal direction, and adds positional encoding to each sequence; For each traversal direction, a forward state space model and a reverse state space model are configured. The forward sequence is input into the forward state space model and the reverse sequence is input into the reverse state space model. Sequence dependency modeling is performed on each model to obtain the bidirectional modeling output features for that traversal direction. Both the forward state space model and the reverse state space model are sequence modeling networks constructed based on discrete state space equations, and they share network parameters. The omnidirectional aggregation module calculates a weight coefficient for the output features of each traversal direction using a convolutional layer with a kernel size of 1×1×1 and an activation function; it determines the traversal direction corresponding to the largest weight coefficient as the main direction, and performs a weighted summation of the output features of the other traversal directions to obtain supplementary aggregation features; then it fuses the main direction features with the supplementary aggregation features to obtain global context enhancement features.

8. A CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 4, characterized in that, The steps of upsampling recovery and predictive probability map generation are performed by the constructed 3D convolutional decoder and the terminal classification layer. A 3D convolutional decoder is constructed in cooperation with the 3D convolutional encoder. The 3D convolutional decoder is composed of multiple upsampling modules and convolutional modules stacked together to gradually recover the spatial resolution of global context enhancement features and finally output a full-size feature map with the same size as the multi-channel input tensor space. The full-size feature map is input into the terminal classification layer, which is a convolutional layer with the number of output channels equal to the total number of preset tooth categories. Its output is processed by a normalized exponential function to generate a predicted probability map of each voxel belonging to each tooth category. A joint loss function, which is a weighted combination of a multi-class Dessian loss function and a cross-entropy loss function, is used to calculate the error between the predicted probability map and the true label, and gradient backpropagation is performed based on this error to update the model parameters; wherein, the multi-class Dessian loss function is a pre-set weight coefficient for each tooth category.

9. A CBCT image tooth segmentation method based on Mamba and multimodal fusion according to claim 1, characterized in that, The prediction of the CBCT image to be segmented is as follows: Obtain the CBCT image to be segmented and its paired case text data; perform the same preprocessing as in step 1 on the CBCT image to be segmented to obtain the multi-channel input tensor to be segmented; the preprocessing includes jaw region cropping, spatial coordinate map generation, local contrast enhancement and channel dimension stitching; The multi-channel input tensor to be segmented and the case text data are input into the trained tooth segmentation model to obtain the predicted probability map of the CBCT image to be segmented. The tooth category corresponding to the maximum value in the channel dimension of the predicted probability map is used as the segmentation label for each voxel, and the tooth segmentation result is output.

Images