A method and system for automatic detection of esophageal foreign bodies

Abstract

The present disclosure provides an esophageal foreign body automatic detection method and system, comprising: acquiring a CT image of a pre-processed esophageal region; based on a three-dimensional convolutional neural network model, cooperating with an anatomical space attention mask, detecting the esophageal region of the CT image, and outputting a minimum three-dimensional bounding box; in the minimum three-dimensional bounding box region, through a feature pyramid network, using the feature pyramid network, extracting multi-scale features, sequentially passing through channel attention enhancement and spatial attention enhancement, detecting and outputting a foreign body candidate box and a foreign body category; for the foreign body in the foreign body candidate box, using a three-dimensional segmentation network for fine segmentation, outputting a foreign body probability map, and performing post-processing on the foreign body probability map to output a foreign body three-dimensional mask; and calculating clinical parameters according to the foreign body three-dimensional mask. The method of the present disclosure can realize a single CT image processing time of not more than 30 seconds; the foreign body detection sensitivity is not less than 90%, and the error of the distance from the incisor is not more than 2 mm.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method and system for automatic detection of esophageal foreign bodies. Background Technology

[0002] Esophageal foreign body aspiration is a common clinical emergency, frequently occurring in children, the elderly, and those who eat hastily. Examples include children swallowing coins or batteries, and adults swallowing fish bones or bone fragments. If esophageal foreign bodies are not detected, located, and assessed promptly and accurately, they can lead to complications such as esophageal mucosal damage, perforation, and infection, seriously threatening the patient's life and health. Therefore, accurate, rapid, and precise detection of esophageal foreign bodies is crucial for clinical treatment.

[0003] Currently, the clinical detection of esophageal foreign bodies mainly relies on imaging and endoscopic examinations, combined with human experience for judgment. Traditional endoscopy allows direct observation of the esophagus, but it is an invasive procedure with risks such as nausea, pain, and even mucosal bleeding. It is also time-consuming, highly dependent on human experience, and unsuitable for infants and young children who cannot cooperate with the examination. Furthermore, it may lead to missed diagnoses due to the limited endoscopic field of view. Prior art 1 (Application No.: 202310983168.8, Application Date: 2023.08.04) discloses a medical image processing method and system for assisting in the removal of fish bones from the esophagus. The method includes: determining the reliability of the binarized image corresponding to the sliding window based on the number of maxima in the binarized image under a preset size parameter; dividing the binarized image into several sub-images based on the reliability of the binarized image to determine the proportion of high brightness corresponding to each sub-image; calculating the selection degree corresponding to each sub-image based on the reliability and the proportion of high brightness corresponding to the sub-image; and determining the enhancement coefficient corresponding to each pixel based on the selection degree corresponding to the sub-image to enhance the corresponding pixels. Although this solution improves the efficiency and accuracy of esophageal fish bone removal through a medical image processing method for assisting in the removal of fish bones from the esophagus, it still cannot accurately obtain the three-dimensional information of the esophageal foreign body.

[0004] X-ray planar imaging can display high-density foreign bodies, but it cannot identify low-density foreign bodies that can be detected by X-rays, such as fish bones and bone fragments, resulting in a low detection rate. Furthermore, two-dimensional X-ray imaging cannot determine the depth of foreign body embedment or its relationship with surrounding tissues, making it difficult to support the formulation of subsequent surgical plans. Prior art 2 (application number: 109130489, application date: 2020.09.04) discloses a throat foreign body identification system and method, comprising: a throat foreign body identification device having a learning identification mode and an auxiliary identification mode, the auxiliary identification mode being used to realize the throat foreign body identification function; an operating device connected to the throat foreign body identification device for user operation to communicate with the throat foreign body identification device and switch the throat foreign body identification device into the learning identification mode or the auxiliary identification mode; a data source device connected to the operating device and storing at least one reference X-ray image data and at least one original X-ray image data for the operating device to read, each reference X-ray image data and each original X-ray image data including red light image data, green light image data and blue light image data; and a display device having an image display function and connected to the operating device. The proposed method uses X-rays for detection, but it still cannot obtain information on the depth of the foreign object in the esophagus, and manual annotation is still required, resulting in low efficiency and accuracy in interpretation.

[0005] CT scans can provide high density resolution and three-dimensional localization capabilities in the detection of esophageal foreign bodies, enabling multi-dimensional tomographic imaging. However, the traditional manual image interpretation mode mainly relies on human subjective experience, which is greatly affected by fatigue, experience differences, and the complexity of the condition. Single-person image interpretation is prone to misdiagnosis, while double-person review will prolong the diagnosis time and may also lead to problems such as missed detection of small foreign bodies and inaccurate localization.

[0006] Therefore, the common technical problems in the existing technology include: 1. low efficiency of manual inspection; 2. easy to miss small foreign objects; 3. low accuracy of three-dimensional positioning. The above technical problems have become the technical problems that urgently need to be solved in this field. Summary of the Invention

[0007] In view of this, the present disclosure provides a method and system for automatic detection of esophageal foreign bodies, which solves the technical problems existing in the prior art: 1. low efficiency of manual detection; 2. easy to miss small foreign bodies; 3. low accuracy of three-dimensional positioning.

[0008] One aspect of this disclosure provides a method for automatic detection of esophageal foreign bodies, comprising:

[0009] The original CT image of the esophagus region is acquired, and an original DICOM format image sequence is generated. The original DICOM format image sequence is preprocessed to obtain a preprocessed CT image. The preprocessing includes: transforming the anisotropic original voxel resampling into an isotropic voxel resolution; adjusting the window width and window level of the original CT image to a soft tissue window; and normalizing the HU value of the CT image to uniformly map it to the [0,1] interval.

[0010] Based on a three-dimensional convolutional neural network model, and in conjunction with an anatomical spatial attention mask, the esophageal region is detected in the preprocessed CT image, and the minimum three-dimensional bounding box of the esophagus in the CT image is output.

[0011] Within the minimum 3D bounding box region, a feature pyramid network is used to extract multi-scale features from the CT image within the region through convolution operations, resulting in a fused multi-scale feature map. This fused multi-scale feature map is then sequentially processed through channel attention enhancement and spatial attention enhancement to perform pixel-level foreign object detection within the esophageal region. Anchor-Free detection heads output candidate foreign object boxes and the foreign object category. Specifically, channel attention enhancement involves performing global average pooling, fully connected layers, and activation on the fused multi-scale feature map to generate a channel weight map. This channel weight map is then multiplied and fused with the fused multi-scale feature map channel-wise to obtain a channel-enhanced feature map. Spatial attention enhancement involves performing global max pooling and global average pooling on the channel-enhanced feature map, followed by convolution and sigmoid activation to generate a spatial weight map. This spatial weight map is then multiplied and fused with the channel-enhanced feature map position-wise to obtain a double-enhanced feature map.

[0012] The foreign object categories include at least fish bones, bone fragments, and metallic foreign objects; the loss function during the detection process is: , ;in, To detect the total loss, For the focus of loss, For balance coefficient, To compare the losses;

[0013] For the foreign objects in the candidate bounding box, a 3D segmentation network is used for fine segmentation, outputting a foreign object probability map. The foreign object probability map is then post-processed to output a 3D mask of the foreign objects. The post-processing includes a conditional random field algorithm or morphological closing operation. The 3D segmentation network includes a main segmentation network and an edge detection auxiliary supervision branch. The total loss function used in the segmentation process is... for: ;in, Total loss due to primary segmentation. To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2;

[0014] The total loss of the main segment The function is: ;in, To divide the total loss, For Dice's loss; For boundary loss; These are the Dice Loss weights, and their values ​​are the Dice loss weights. The boundary loss weight is set to 0.3.

[0015] The edge detection auxiliary loss The calculation uses the binary cross-entropy loss function;

[0016] Based on the 3D mask of the foreign body, clinical parameters are calculated, including location parameters, geometric parameters, and impaction risk. The location parameters are the distance of the foreign body from the incisors and the segmented position of the foreign body in the esophagus, obtained through calibration and mapping between the CT coordinate system and the endoscopic coordinate system. The geometric parameters include the maximum diameter of the foreign body, the volume of the foreign body, and the CT value of the foreign body. The maximum diameter of the foreign body is calculated by taking the distance of the maximum spatial diagonal based on the 3D point cloud of the foreign body. The volume of the foreign body is calculated by performing voxel-level segmentation on the foreign body target in the CT image, counting the number of effective voxels, obtaining the total number of effective voxels and the volume of a single voxel, and then using the total number of voxels... The volume of the foreign object is obtained by calculating the volume of a single voxel; the CT value refers to the mean ± standard deviation of the voxel HU values ​​within the region, and the calculation method of the CT value includes: calculating the arithmetic mean and standard deviation of the CT values ​​of all voxels within the foreign object mask; the embedding risk includes low-level risk, medium-level risk, and high-level risk, and the calculation method of the embedding risk includes: extracting the sharpness of the foreign object based on the curvature change of the mask surface of the three-dimensional mask of the foreign object; extracting the irregularity of the foreign object based on the volume ratio of the volume of the foreign object to the volume of the minimum circumscribed sphere of the foreign object; extracting the abnormal CT value region features of the tissue surrounding the foreign object; inputting the sharpness of the foreign object, the irregularity of the foreign object, and the abnormal CT value region features of the tissue surrounding the foreign object into a pre-trained evaluation model, and outputting the embedding risk level.

[0017] Optionally, the isotropic voxel resolution is 0.5 mm × 0.5 mm × 0.5 mm - 2.0 mm × 2.0 mm × 2.0 mm.

[0018] Optionally, the step of detecting the esophageal region in the preprocessed CT image based on a three-dimensional convolutional neural network model, in conjunction with an anatomical spatial attention mask, and outputting the minimum three-dimensional bounding box of the esophagus in the CT image includes the following steps:

[0019] Based on the preprocessed CT images, and according to the anatomical prior that the esophagus is located behind the trachea and in front of the spine, an anatomical spatial attention mask is generated.

[0020] A three-dimensional convolutional neural network model is used to downsample the preprocessed CT images layer by layer, extract features, and obtain the bottom feature map.

[0021] The anatomical space attention mask is fused with the bottom feature map to obtain the fused bottom feature map;

[0022] A three-dimensional convolutional neural network model is used to upsample the fused low-level feature map layer by layer to obtain the esophageal probability map.

[0023] After binarizing the esophageal probability map, the largest connected component is selected, the smallest bounding box of the largest connected component is calculated, and the smallest three-dimensional bounding box of the esophagus in the CT image is output.

[0024] Optionally, the three-dimensional convolutional neural network model is a 3D U-Net model or a 3D Mask R-CNN model.

[0025] Optionally, the Anchor-Free detection head is a CenterNet probe or a YOLOv8 probe.

[0026] Optionally, the main segmentation network is a 3D U-Net++ or V-Net;

[0027] The edge detection auxiliary supervision branch is the Canny edge detection branch.

[0028] Optionally, the evaluation model is a random forest model or a logistic regression model.

[0029] Optionally, the method further includes:

[0030] A foreign body detection heatmap is generated using the Grad-CAM method. The three-dimensional mask of the foreign body is then superimposed on the original CT image to output a structured report.

[0031] Another aspect of this disclosure provides an automatic esophageal foreign body detection and localization system, comprising: an automatic esophageal region localization module, a small target foreign body enhanced detection module, a foreign body boundary fine segmentation module, and a clinical parameter calculation and risk assessment module; wherein,

[0032] The esophageal region automatic localization module is used to input the original CT image, preprocess the original CT image to obtain the preprocessed CT image, and based on the three-dimensional convolutional neural network model, in conjunction with the anatomical space attention mask, to detect the esophageal region in the preprocessed CT image and output the minimum three-dimensional bounding box of the esophagus in the CT image.

[0033] The small target foreign object enhancement detection module is coupled to the esophageal region automatic localization module. It is used to input the minimum 3D bounding box, and through a feature pyramid network, performs convolution operations on the CT image within the minimum 3D bounding box region to extract multi-scale features. These features are then sequentially enhanced through channel attention and spatial attention to perform pixel-level detection of foreign objects within the esophageal region. The Anchor-Free detection head outputs candidate foreign object boxes and foreign object categories. The loss function during the detection process... for: , ;in, To detect the total loss, For the focus of loss, For balance coefficient, To compare the losses between the two;

[0034] The foreign object boundary fine-grained segmentation module is coupled to the small target foreign object enhanced detection module. It is used to input the foreign object candidate box, perform fine segmentation of the foreign objects in the candidate box using a 3D segmentation network, output a foreign object probability map, and perform post-processing on the probability map to output a 3D mask of the foreign object. The post-processing includes a conditional random field algorithm or morphological closing operation. The 3D segmentation network includes a main segmentation network and an edge detection auxiliary supervision branch. The total loss function used in the segmentation process is... for: ;in, Total loss due to primary segmentation. To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2;

[0035] The total loss of the main segmentation The function is: ;in, For Dice's loss; For boundary loss; These are the Dice Loss weights, and their values ​​are the Dice loss weights. The boundary loss weight is set to 0.3.

[0036] The edge detection auxiliary loss The calculation uses the binary cross-entropy loss function;

[0037] The clinical parameter calculation and risk assessment module is coupled to the foreign body boundary fine segmentation module. It is used to input the three-dimensional mask of the foreign body and calculate the position parameters, geometric parameters, and impaction risk based on the three-dimensional mask of the foreign body. The position parameters are the distance of the foreign body from the incisors and the segmented position of the foreign body in the esophagus. The geometric parameters include the maximum diameter of the foreign body, the volume of the foreign body, and the CT value of the foreign body. The impaction risk includes low-level risk, medium-level risk, and high-level risk.

[0038] Optionally, it also includes an interpretability output module, which is coupled to the clinical parameter calculation and risk assessment module and is used to input the calculation and assessment results output by the clinical parameter calculation and risk assessment module to generate a foreign body detection heatmap using the Grad-CAM method.

[0039] Compared with the prior art, the method and system for automatic detection of esophageal foreign bodies provided in this disclosure achieve at least the following beneficial effects:

[0040] First, the method disclosed herein inputs CT images of the esophagus region into a deep learning-based automatic detection system for esophageal foreign bodies in CT images for automatic detection and interpretation. This solves the problem of low efficiency and reliance on doctors' experience in interpreting CT images of esophageal foreign bodies, and significantly improves the efficiency of emergency image interpretation. It can achieve a single CT image processing time of no more than 30 seconds.

[0041] Secondly, the method disclosed herein automatically locates the esophageal region using a three-dimensional convolutional neural network (3D CNN) and combines it with a multi-scale feature fusion target detection algorithm to achieve pixel-level detection and segmentation of foreign objects, solving the problem of easily missing small foreign objects; the foreign object detection sensitivity is not less than 90%, which is 15%-20% higher than manual interpretation;

[0042] Third, this method is based on CT images and optimization of a three-dimensional segmentation network and boundary loss function. It can accurately segment detected foreign objects, calculate the location of the foreign object (upper / middle / lower esophagus, distance from incisors), geometric parameters (maximum diameter, volume, CT value), and assessment of the risk of embedment, thus improving the accuracy of three-dimensional localization. It can achieve an error of no more than 2mm in the distance from incisors.

[0043] Fourth, in the system disclosed herein, the esophageal region automatic localization module, based on a three-dimensional convolutional neural network, can reduce the search space by 80%; the small target foreign body enhanced detection module, based on a multi-scale feature fusion network and attention mechanism, can achieve pixel-level detection of foreign bodies within the esophageal region; the foreign body boundary fine segmentation module can achieve fine segmentation of foreign bodies, improving localization accuracy; and the clinical parameter calculation and risk assessment module can obtain the precise location parameters, geometric parameters, and embedding risk of foreign bodies, which can assist in rapid manual decision-making, provide precise navigation for endoscopic examinations, and shorten operation time.

[0044] Of course, any product implementing this disclosure does not necessarily need to achieve all of the technical effects described above at the same time.

[0045] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0047] Figure 1 This is a flowchart of an automatic method for detecting foreign bodies in the esophagus provided in this disclosure;

[0048] Figure 2 This is a flowchart of another method for automatic detection of esophageal foreign bodies provided in this disclosure;

[0049] Figure 3 This is a flowchart of the entire process of an automatic method for detecting foreign bodies in the esophagus provided in this disclosure;

[0050] Figure 4 This is a flowchart illustrating the combination of an automatic method and system for detecting esophageal foreign bodies provided in this disclosure;

[0051] Figure 5 This is a flowchart illustrating step S2 of an automatic method for detecting esophageal foreign bodies provided in this disclosure;

[0052] Figure 6 This is an overall structural block diagram of another automatic esophageal foreign body detection and positioning system provided in this disclosure. Detailed Implementation

[0053] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0054] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0055] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0056] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0057] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0058] Reference Figures 1-5 This disclosure provides a method for automatic detection of esophageal foreign bodies, comprising:

[0059] S1. Obtain the original CT image of the esophagus region, generate the original DICOM format image sequence, preprocess the original DICOM format image sequence, and obtain the preprocessed CT image. The preprocessing includes: transforming the anisotropic original voxel resampling into isotropic voxel resolution; adjusting the window width and window level of the original CT image to soft tissue window; and normalizing the HU value of the CT image to uniformly map it to the [0,1] interval.

[0060] S2. Based on a three-dimensional convolutional neural network model (3D CNN), and in conjunction with an anatomical spatial attention mask, the esophageal region is detected in the preprocessed CT images, and the minimum three-dimensional bounding box of the esophagus in the CT images is output.

[0061] S3. Within the smallest 3D bounding box region, a Feature Pyramid Network (FPN) is used to extract multi-scale features from the CT image within the smallest 3D bounding box region through convolution operations, resulting in a fused multi-scale feature map. This fused multi-scale feature map is then enhanced sequentially using Channel Attention SENet and Spatial Attention CBAM to perform pixel-level foreign object detection within the esophageal region. Anchor-Free detection heads output foreign object candidate boxes and foreign object categories. Specifically, Channel Attention SENet enhancement involves global average pooling, fully connected layers, and activation on the fused multi-scale feature map to generate channel weight maps. These weight maps are then multiplied and fused with the fused multi-scale feature map channel-wise to obtain the channel-enhanced feature map. Spatial Attention enhancement involves global max pooling and global average pooling along the channel dimensions of the channel-enhanced feature map, followed by convolution and Sigmoid activation to generate spatial weight maps. These weight maps are then multiplied and fused with the channel-enhanced feature map position-wise to obtain the double-enhanced feature map. Foreign object categories include at least fish bones, bone fragments, and metallic foreign objects. The loss function during detection is: , ;in, To detect the total loss, For the focus of loss, For balance coefficient, To compare the losses between the two;

[0062] S4. For foreign objects in the candidate bounding box, a 3D segmentation network is used for fine segmentation, outputting a foreign object probability map. The foreign object probability map is then post-processed to output a 3D mask of the foreign object. This post-processing includes a conditional random field algorithm or morphological closing operation. The 3D segmentation network includes a main segmentation network and an edge detection auxiliary supervision branch. The total loss function used in the segmentation process is... for: ;in, Total loss due to primary segmentation. To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2; the total loss for main segmentation... The function is: ;in, For Dice's loss; For boundary loss; These are the Dice Loss weights, and their values ​​are the Dice loss weights. The boundary loss weight is set to 0.3; the edge detection auxiliary loss... The calculation uses the binary cross-entropy loss function;

[0063] S5. Based on the 3D mask of the foreign body, calculate the clinical parameters, including location parameters, geometric parameters, and impaction risk. Location parameters are the distance of the foreign body from the incisors and the segmented position of the foreign body in the esophagus, obtained through calibration and mapping between the CT coordinate system and the endoscopic coordinate system. Geometric parameters include the maximum diameter of the foreign body, the volume of the foreign body, and the CT value of the foreign body. The maximum diameter of the foreign body is calculated by taking the distance of the maximum spatial diagonal based on the 3D point cloud of the foreign body. The volume of the foreign body is calculated by performing voxel-level segmentation on the foreign body target in the CT image, counting the number of effective voxels, obtaining the total number of effective voxels and the volume of a single voxel, and then using the total number of voxels... The volume of the foreign object is obtained by calculating the volume of a single voxel; the CT value refers to the mean ± standard deviation within the region. The calculation method for the CT value includes: calculating the arithmetic mean and standard deviation of the CT values ​​of all voxels within the foreign object mask; the embedding risk is divided into low-level risk, medium-level risk, and high-level risk. The calculation method for the embedding risk includes: extracting the sharpness of the foreign object based on the curvature change of the mask surface of the 3D mask of the foreign object; extracting the irregularity of the foreign object based on the volume ratio of the volume of the foreign object to the volume of the smallest circumscribed sphere of the foreign object; extracting the abnormal CT value region features of the tissue surrounding the foreign object; inputting the sharpness of the foreign object, the irregularity of the foreign object, and the abnormal CT value region features of the tissue surrounding the foreign object into the pre-trained evaluation model, and outputting the embedding risk level.

[0064] Specifically, in step S1, isotropic resampling (e.g., 1×1×1 mm³) is performed on the CT images (DICOM format) to eliminate inter-slice anisotropy. The processed image voxels are cubes with consistent three-dimensional spatial scale and no inter-slice stretching deformation. Optionally, trilinear interpolation is used to perform isotropic resampling on the CT images. The window width and window level of the CT images are adjusted to the mediastinal window (soft tissue window: 350 / 40 HU). The processed image CT value range is compressed to -135~+215 HU, enhancing the contrast of the esophageal wall and mediastinal soft tissue, while appropriately suppressing bones and air. The HU value of the CT images is normalized (0~1 interval). The processed image has a uniform grayscale distribution, and the input to the deep learning model is standardized, accelerating convergence. In the preprocessed CT images, structures such as the esophagus, trachea, and spine are clearly distinguishable, foreign objects (fish bones) appear as high-density shadows, and soft tissue contrast is good.

[0065] Specifically, in step S2, anatomical structures such as the trachea, spine, and mediastinum are first automatically segmented based on CT images. Based on the anatomical prior that the esophagus is located behind the trachea and in front of the spine, a 3D spatial attention mask is generated. This mask assigns high weights (close to 1) to the potential esophageal region and low weights (close to 0) to other regions. This mask is then multiplied voxel-by-voxel with the lowest-resolution feature map of the 3D CNN encoder, causing the network to focus on the esophageal region. Furthermore, the attention mask is concatenated as an additional channel with the upsampled features at the skip connections of the decoder to enhance edge details. The network predicts accurate attention masks through supervised learning. Finally, under attention guidance, the minimum 3D bounding box of the esophagus is output, reducing the search space for subsequent foreign body detection by 80%.

[0066] Specifically, step S3 includes:

[0067] Step S31: Input the CT image of the esophageal minimum bounding box region and extract multi-scale features using a Feature Pyramid Network (FPN): First, a bottom-up path is used to extract feature maps C1~C5 through 5-level convolutional downsampling. The spatial resolution is halved layer by layer (1 / 2, 1 / 4, 1 / 8, 1 / 16), the number of channels is doubled layer by layer (32, 64, 128, 256, 512), and the receptive field is expanded layer by layer to capture texture details, local shapes, component structures, overall morphology, and global semantics. For example, in one embodiment, the size of the receptive field in feature map C1 is 7×7×7. The C1 feature map is suitable for detecting extremely small fish bones (<1mm); the C2 feature map has a receptive field size of 15×15×15, suitable for detecting general fish bones (1-2mm); the C3 feature map has a receptive field size of 31×31×31, suitable for detecting bone fragments (2-5mm); the C4 feature map has a receptive field size of 63×63×63, suitable for detecting metallic foreign objects (≥5mm); and the C5 feature map has a receptive field size of 127×127×127, suitable for locating large foreign objects. Then, a top-down approach is adopted, using upsampling (2x) and lateral connections to progressively pass high-level semantic information to lower levels: C5 is upsampled after 1×1×1 convolution dimensionality reduction, added to C4, and then convolved again by 3×3×3 to generate P4; this process is repeated to generate P3, P2, and P1 sequentially. The final output is multi-scale feature maps P1~P5, with the highest resolution P1 (same size as the input) retaining the most detailed information for subsequent foreign object detection. This design enables the network to detect both extremely small foreign objects (<1mm fishbone) and large foreign objects (>5mm metal) simultaneously, exhibiting high sensitivity to foreign objects of different sizes.

[0068] It should be noted that in step S3, the bottom-up path extracts hierarchical features from high resolution (low semantics) to low resolution (high semantics), gradually expanding the receptive field and capturing contextual information at different scales. The input CT image within the smallest 3D bounding box region is used to sequentially generate C1, C2, C3, C4, and C5. The C1 feature map contains detailed features (edges, textures); the C2 feature map contains local structure (foreign object outline); the C3 feature map contains component features (partial areas of the foreign object); the C4 feature map contains semantic features (the foreign object as a whole); and the C5 feature map contains global semantics (context). The top-down path transmits high-level semantic information back to low-level features, enhancing the semantic expressive power of detailed features, and simultaneously fusing and locating details through lateral connections. P5, P4, P3, P2, and P1 are generated sequentially. The feature maps are divided into three types: P5, suitable for detecting and locating large foreign objects; P4, suitable for detecting metallic foreign objects (≥5mm); P3, suitable for detecting bone fragments (2-5mm); P2, suitable for detecting common fish bones (1-2mm); and P1, suitable for detecting extremely small fish bones (<1mm). The content extracted for each size differs: large-scale (P2, P1) features high-resolution details, including the edges, textures, and sharp ends of the foreign object; suitable for small targets (small fish bones, <2mm in diameter); medium-scale (P3) features local structural features, including the local shape and curvature of the foreign object; suitable for medium-sized foreign objects (bone fragments, 2-5mm in diameter); and small-scale (P5, P4) features global semantic features, including the overall outline of the foreign object and its spatial relationship with surrounding tissues; suitable for large foreign objects (metallic foreign objects, >5mm in diameter). After multi-scale feature extraction, the highest resolution P1 (retaining the most detailed information) from P1 to P5 is fed into the subsequent attention enhancement module and detection head for small target foreign object detection. P2~P5 are used internally within the FPN to assist in training and feature transfer, and are not directly output to the detection head.

[0069] Step S32, SENet Channel Attention Enhancement: Channel attention enhancement targets the semantic importance of different channels. The operations employed include global average pooling, fully connected layers, and activation, specifically global average pooling—fully connected layer 1—ReLU activation—fully connected layer 2—Sigmoid activation. The output channel weight vector S is then multiplied and fused with the fused multi-scale feature map F channel by channel weight vector S to obtain the enhanced feature map. In channel attention enhancement, for foreign object response channels (channels sensitive to high-density foreign objects (fish bones, bone fragments), foreign object signals are enhanced and background is suppressed; for edge feature channels (channels sensitive to foreign object boundaries and contours), boundary detection capabilities are improved; for texture feature channels (channels sensitive to foreign object surface textures (such as fish bone patterns), foreign object type identification is assisted; and for irrelevant feature channels (channels sensitive to background structures such as bones, trachea, and blood vessels), background interference is suppressed. Global average pooling refers to global information compression. It performs global average pooling on each channel to obtain a channel description vector, which compresses the spatial information of each channel into a scalar representing the global response intensity of that channel. Fully connected layer 1 reduces dimensionality and computation. ReLU activation introduces nonlinearity. Fully connected layer 2 restores the original number of channels. Sigmoid activation normalizes the weights of each channel to [0-1]. Feature recalibration multiplies the original feature map with the channel weights channel by channel, amplifying important channels (weights close to 1) and suppressing unimportant channels (weights close to 1). For example, channel 1 (foreign object response), the original feature is high response, the channel weight S is 0.95, and after enhancement it is high response (enhancement), the foreign object signal is preserved; channel 2 (edge ​​response), the original feature is medium response, the channel weight is 0.85, and after enhancement it is medium-high response (enhancement), the edge signal is preserved; channel 3 (background response), the original feature is high response, the channel weight is 0.10, and after enhancement it is low response (suppression), the background signal is suppressed; channel 4 (noise response), the original feature is medium response, the channel weight is 0.05, and after enhancement it is extremely low response (suppression), the noise is suppressed. That is, in this method, channel attention enhancement adopts the SENet module: first, global average pooling is performed on the input feature map to compress the spatial information of each channel into a scalar to obtain the channel description vector; then, the non-linear dependencies between channels are learned through two fully connected layers (dimensionality reduction ratio r=16) and ReLU activation; then, channel weights are generated through Sigmoid activation; finally, the original feature map is multiplied with the channel weights channel by channel to enhance important channels (foreign object response, edge features) and suppress irrelevant channels (skeleton, atmospheric background).

[0070] In one embodiment, global average pooling is performed on P1 to compress the global response of each channel into a 256-dimensional vector; channel weights are generated by passing through fully connected layer 1 (compression ratio r=16, dimensionality reduced to 16), ReLU activation, fully connected layer 2 (compression ratio r=16, restored to 256), and Sigmoid activation; P1 is multiplied by the channel weights s channel by channel to enhance important channels (foreign object response, edge features) and suppress irrelevant channels (skeleton, air), as shown in the example in Table 1.

[0071] Table 1

[0072]

[0073] Step S33, CBAM Spatial Attention Enhancement: Spatial attention enhancement targets the importance of different spatial locations. The operations include: channel pooling (global max pooling and global average pooling along the channel dimension) – concatenation – 3×3×3 convolution – Sigmoid activation on the obtained enhanced feature maps; outputting a spatial weight map, multiplying and fusing the enhanced feature map positionally (broadcast to each channel) with the spatial weight map to obtain a doubly enhanced feature map. In spatial attention enhancement, for foreign object spatial locations (continuous voxel regions where the foreign object is located), the response of the foreign object region can be enhanced; for boundary regions (voxels at the boundary between the foreign object and the background), the boundary segmentation accuracy can be improved; for background regions (esophageal lumen, surrounding soft tissue, etc.), the background response can be suppressed. Spatial attention enhancement is performed on the channel-enhanced feature map as input. Two pooling operations are applied along the channel dimension: global max pooling to highlight the strongest response and global average pooling to highlight regions with high average response. These are then concatenated. After 3D convolution and sigmoid activation, a spatial weight map M is generated. The 3D convolution learns the weight distribution of spatial locations and captures local spatial dependencies. The sigmoid function normalizes the weight of each spatial location to between 0 and 1. The channel-enhanced feature map is multiplied positionally by the spatial weights to enhance the response at the location of the foreign object and suppress the background region. This generates a double-enhanced feature map. It should be noted that all channels share the same spatial weights. For example, in spatial weighting, the original feature of the foreign object's central region is high response, with a spatial weight of 0.92. After enhancement, it becomes high response (enhancement), thus strengthening the foreign object region. The original feature of the foreign object's edge region is medium response, with a spatial weight of 0.85. After enhancement, it becomes medium-high response (enhancement), preserving the boundary region. The original feature of the esophageal air is low response, with a spatial weight of 0.10. After enhancement, it becomes extremely low response (suppression), suppressing the background. The original feature of the surrounding soft tissue is low response, with a spatial weight of 0.05. After enhancement, it becomes extremely low response (suppression), suppressing the background. Specifically, spatial attention enhancement uses the CBAM spatial attention module: global max pooling and global average pooling are performed on the enhanced feature maps along the channel dimension to generate two spatial feature maps, highlighting the region with the strongest response and the region with a high average response, respectively. These two maps are concatenated and then convolved with a sigmoid activation to generate a spatial weight map. Finally, the feature map and the spatial weight map are multiplied positionally (weights are broadcast to all channels), achieving enhancement of the foreign object's spatial location and suppression of the background region.

[0074] Through the aforementioned dual attention enhancement, the foreign object-related features are significantly enhanced in both the channel and spatial dimensions, background interference is effectively suppressed, and the ability of the subsequent detection head to identify small target foreign objects in the esophagus is significantly improved.

[0075] Step S34, CenterNet Detector Head Detection: In step S3, a loss function is used for multi-task loss optimization during detection to balance the accuracy of foreign object classification and bounding box localization. The specific formula is: , ;in, To detect the total loss, For focus losses, For balance coefficient, This is the Generalized IoU Loss.

[0076] The specific usage of the loss function is as follows: Focal Loss is calculated by combining the probabilities of the foreign object category predicted by the Anchor-Free detector with the true category, and GIoU Loss is calculated by combining the predicted 3D bounding box with the true bounding box. The two are then added together with weights λ (ranging from 0.5 to 1.0) to obtain the total loss. By updating the parameters of the entire detection module (FPN, attention, and detection head) through backpropagation, the model can be guided to simultaneously optimize the accuracy of foreign object classification and localization.

[0077] Specifically, during detection by the detection head, the double-enhanced feature map is input into three parallel branches:

[0078] The center point heatmap branch outputs the probability of the foreign object's center point after a 1×1×1 convolution and a sigmoid function. To address the imbalance between positive and negative samples, Focal Loss is used as the loss function for this branch. The Focal Loss function is as follows:

[0079] ;

[0080] in The actual labels are (the center point is 1, and the rest are 0). , where is the modulation factor; N is the total number of samples.

[0081] The 3D dimension branch outputs the length, width, and height of the foreign object to predict its 3D physical dimensions. This branch uses the L1 Loss function as the loss function to calculate the error between the predicted and actual dimensions. The L1 Loss function is: ;in, To predict the size, This is the actual size. This represents the number of all positive samples (center points of foreign objects).

[0082] The angle branch outputs the principal axis orientation angle of the foreign object, which corresponds to the principal axis orientation angle of the foreign object in three-dimensional space. This branch uses the L1 Loss function as the loss function, which is: ;in, For the prediction angle, From a realistic perspective; This represents the number of all positive samples (center points of foreign objects).

[0083] Therefore, the total loss function in this step is: ;in, , .

[0084] To further remove duplicate bounding boxes and retain the foreign object side results with high confidence, post-inference processing is performed as follows: Input the heatmap, size prediction, and angle prediction results; perform max pooling on the heatmap to filter out candidate center points with confidence higher than a set threshold (0.3); read the size and angle of the candidate center points to generate 3D candidate boxes; sort them from high to low confidence; for each candidate box, merge overlapping boxes using non-maximum suppression (NMS, IoU threshold 0.5), remove candidate boxes with lower confidence, and output the final list of foreign object candidate boxes, where IoU = intersection volume / union volume.

[0085] Through the above steps, high sensitivity detection (≥90%) and high positioning accuracy (distance error from incisors ≤2mm) of small target foreign objects in the esophagus are achieved.

[0086] It should be noted that the number of foreign object candidate boxes depends on the actual number of foreign objects present in the CT image: one candidate box is output for a single foreign object, and multiple candidate boxes are output for multiple foreign objects. The detection head first outputs multiple candidate boxes (including overlapping boxes), then removes redundancy through non-maximum suppression (NMS), and finally retains the candidate box with the highest confidence and no overlap. Each candidate box corresponds to an independent foreign object target, including center point coordinates, three-dimensional size, and angle information.

[0087] It should be noted that in this method, all deep learning modules (including the 3D segmentation network) are trained iteratively. This means that the training dataset is traversed multiple times, with the loss function calculated and backpropagation used to update network parameters each time, until the model converges or reaches a preset stopping condition. After each round, the model performance is evaluated on the validation set. If performance improves, the model checkpoint is saved; if the early stopping condition is met (loss does not decrease for 20 consecutive rounds), the model terminates early. The total loss function is dynamically constructed during training. In each iteration, the loss value is dynamically calculated based on the model's predictions and the ground truth labels. During the inference phase, only forward propagation is performed; no loss is calculated, and no backpropagation is performed.

[0088] Specifically, in step S4, the loss function construction process includes: inputting the predicted probability map (model output) + the ground truth mask (manually labeled), calculating the Dice loss function and the Boundary IoU loss, and after weighted fusion, obtaining the total loss function for the main segmentation: ;in, To divide the total loss, For Dice Loss; For boundary loss; This is the Dice Loss weight, with a value of 0.7. The boundary loss weight is set to 0.3.

[0089] The main segmentation loss function optimizes all trainable parameters of the 3D segmentation network (3D U-Net++ or V-Net), including the weights and biases of all convolutional layers in the encoder and decoder, the scaling and translation factors of the batch normalization layers, and the convolutional parameters of the edge detection auxiliary branches during training. These parameters are iteratively updated through backpropagation to ensure that the network output foreign object mask achieves both high Dice coefficients and high boundary IoU. The loss function constrains the model's learning direction, making the model's predictions closer to the ground truth labels. Specifically, the Dice Loss function is constrained by region overlap, maximizing the overlap between the predicted and ground truth masks (maximizing the Dice coefficient); the Boundary IoU Loss function is constrained by boundary accuracy, maximizing the overlap between the predicted and ground truth boundaries (maximizing boundary IoU). When =0, it means that the predicted mask and the real mask almost completely overlap (Dice ≈ 1), and the predicted boundary and the real boundary almost completely coincide (Boundary IoU ≈ 1).

[0090] In one embodiment, during the training of the main segmentation total loss function, 1200 annotated CT images of esophageal foreign bodies are prepared as training samples, including 600 cases of fish bones, 400 cases of bone fragments, and 200 cases of metallic foreign bodies. The data is divided into a 70% training set (840 cases), a 15% validation set (180 cases), and a 15% test set (180 cases), and random rotation (±10°), random scaling (0.9-1.1 times), and elastic deformation (…). =2, Data augmentation was performed with a resolution of 0.5.

[0091] In this method, the 3D segmentation network is trained iteratively. During training, each iteration dynamically constructs a loss function based on the model's predictions and the ground truth annotations to constrain the model's learning direction. The training iteration process involves uniform initialization of network parameters using Kaiming and employing the AdamW optimizer (initial learning rate 1×10⁻⁶). -4 Weight decay 1×10 -5 The batch size is 4. In each iteration, the forward propagation outputs a foreign object probability map, which is compared with the real mask to calculate the loss value. The network parameters are then updated through backpropagation. Cosine annealing learning rate scheduling (periodic) is used. =50), early stopping is triggered when the validation set loss does not decrease for 20 consecutive rounds. Specific training stopping conditions include: automatically stopping after reaching the maximum number of rounds after training for 300 rounds; early stopping is triggered when the validation set loss does not decrease for 20 consecutive rounds and recovers to the optimal round parameters; and stopping when the validation set Dice coefficient no longer increases for 10 consecutive rounds (increase <0.001), reaching performance saturation.

[0092] Loss function construction: Dynamically construct the combined loss function during training. .in: ; Constrain the region overlap between the predicted mask and the real mask to maximize the Dice coefficient; where, For the prediction probability graph, the first The probability value of a voxel. For the first in the real mask The label of the individual element (0 or 1); It is a prime number in the whole. ; Constrain the overlap between the predicted boundary and the true boundary to maximize the BoundaryIoU, where, To predict the boundary voxel set of the mask, The boundary voxel set of the real mask.

[0093] By minimizing the combined loss, the network simultaneously learns high region overlap (Dice ≥ 0.90) and high boundary accuracy (Boundary IoU ≥ 0.86), ensuring that the output foreign object mask completely covers the foreign object region and has smooth, continuous boundaries. After training, the model parameters are fixed (saved as a .pth file). During inference, only forward propagation is performed; the loss is no longer calculated, and the foreign object probability map is directly output. The final foreign object mask is obtained through CRF post-processing.

[0094] During the training phase, an edge detection auxiliary supervision branch is also included. This branch acts as an auxiliary supervision signal and is trained jointly with the main segmentation network. During the inference phase, this branch is removed, and only the output of the main segmentation network is used to ensure inference speed. Employing an edge detection auxiliary supervision branch enhances boundary awareness, forcing the main segmentation network to focus on the fine structures of foreign object edges, thus improving the segmentation accuracy for slender foreign object boundaries such as fish bones. It can also alleviate class imbalance, as boundary voxels constitute only a very small percentage (typically <5%), and independent supervision prevents the main network from ignoring edge regions. Furthermore, it improves generalization, as auxiliary supervision can be seen as a regularization method, reducing overfitting and allowing the model to better adapt to different shapes of foreign object boundaries. Finally, it improves boundary continuity, especially for the tips and edges of slender foreign objects (such as fish bones), reducing segmentation breakage.

[0095] The edge detection auxiliary supervision branch receives its input from the feature map of the highest resolution layer of the main segmentation network decoder; after convolution and other operations, it outputs an edge probability map, where each voxel value represents the probability that the location belongs to the boundary of a foreign object; it then inputs a manually labeled foreign object mask, performs 3D Canny edge detection, and outputs the true edge labels. The edge loss is calculated using binary cross-entropy. ;in, The first predicted marginal branch The probability that an individual element belongs to the boundary; The actual edge label (0 or 1); The total number of prime numbers;

[0096] Therefore, the total loss function for: ;in, Main segmentation loss, To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2.

[0097] The 3D segmentation network also includes an edge detection auxiliary supervision branch, which is only used during the training phase. Its input is the feature map of the highest resolution layer of the main segmentation network decoder, which is then sequentially processed by 3×3×3 convolutions (32→16), batch normalization, ReLU, 1×1×1 convolutions (16→1), and Sigmoid activation, outputting an edge probability map. Each voxel value represents the probability that the location belongs to the foreign object boundary. The true edge labels are generated by a 3D Canny edge detection algorithm using a manually labeled foreign object mask, including Gaussian smoothing, gradient calculation, non-maximum suppression, and double thresholding, outputting binary edge labels. The loss function for the edge detection branch uses binary cross-entropy loss: weighted by... Added to the total loss, of which: , .

[0098] Through joint training, the gradients of the edge branches are passed to the main segmentation network via shared features, forcing the network to focus on the voxels at the edges of foreign objects, thus improving the accuracy and continuity of boundary segmentation. During the inference phase, the edge detection branch is removed, and only the main segmentation network is retained for forward propagation to ensure inference speed.

[0099] After binarization, the probability map output by the segmentation network may exhibit the following problems: isolated noise points, manifested as scattered small-area false positives, mainly caused by network misjudgment; micro-holes, manifested as voids inside the mask, mainly caused by partial volumetric effects; jagged edges, manifested as discontinuous edges, mainly caused by resolution limitations; and over-segmentation / under-segmentation, manifested as inaccurate boundaries, mainly caused by insufficient feature extraction. To address these issues, a Conditional Random Field (CRF) algorithm or morphological closing operation is used for post-processing.

[0100] Conditional Random Fields (CRFs) utilize the spatial consistency between voxels to optimize segmentation results by modeling the relationships between adjacent voxels, resulting in smoother boundaries and reduced noise. The objective of a CRF is to minimize the following energy function. , ;

[0101] in, As a unit of potential energy, it comes from network prediction, encourages labels to be consistent with predictions, and maintains the original information of network predictions; ; For the network prediction of the first The probability that an individual element belongs to a foreign object; For binary potential energy, spatial smoothing term, encourages consistent labels for adjacent voxels; ; External core Its purpose is to encourage voxel labels with similar colors / grayscale to be the same; its expression is: , The appearance kernel weight represents the intensity of the influence of grayscale similarity. In one embodiment... =10.0; The standard deviation of spatial distance represents the spatial distance weights (voxels). In one embodiment, =3.0; The standard deviation of grayscale distance represents the grayscale difference weight (HU value). In one embodiment, =10.0; smoothing kernel ( The purpose of ) is to encourage spatially adjacent voxels to have the same label, and its expression is: , The smoothing kernel weights, representing the strength of the effect of spatial smoothing, are used in one embodiment. =3.0; The standard deviation of the smoothing kernel represents the spatial range of the smoothing kernel. In one embodiment, =3.0; For compatibility functions, the Potts model is used in one embodiment, with a penalty of 1 for different labels and 0 for the same label. After 10 iterations of mean-field optimization, isolated noise points are eliminated, tiny holes inside the mask are filled, and a smoothed foreign object mask is output.

[0102] Morphological closing is a combined operation of dilation and erosion on a binary mask, used to fill tiny holes within the mask, smooth boundaries, and connect neighboring regions. First, dilation is performed. For each voxel, its neighborhood (within the structuring element, which uses a spherical kernel with a radius of 2 voxels) is checked. If at least one voxel in the neighborhood has a value of 1, that voxel is set to 1. Dilation expands the mask area, filling small holes and connecting broken regions. Then, erosion is performed. For each voxel, its neighborhood (within the structuring element) is checked. If all voxels in the neighborhood have a value of 1, that voxel remains 1; otherwise, it is set to 0. Erosion restores the original mask size and smooths boundaries. This scheme is computationally fast (<0.1 seconds / example) and suitable for scenarios with high real-time requirements.

[0103] In this method, post-processing employs Conditional Random Fields (CRF) or morphological closing operations to optimize the probability map output by the segmentation network. Both post-processing methods effectively eliminate isolated noise, fill holes, and smooth boundaries, resulting in a final foreign body mask with better spatial continuity and clinical usability. It should be noted that the CRF algorithm has high computational complexity and relatively slow processing time (approximately 1-2 seconds / case), and is highly sensitive to parameters, requiring parameter tuning. However, it significantly improves accuracy (boundary IoU +5-8%), making it suitable for scenarios with high accuracy requirements and ample time, such as clinical diagnostic assistance and research / offline analysis. Morphological closing operations, on the other hand, have low computational complexity and lower parameter sensitivity, requiring only control of the structuring element radius. They offer moderate accuracy improvement (boundary IoU +2-3%), but extremely fast processing time (<0.1 seconds / case), making them suitable for scenarios with high real-time requirements and acceptable accuracy, such as rapid emergency screening and embedded PACS deployments.

[0104] In one embodiment, the foreign object within each candidate box is finely segmented. First, a 64×64×64 sub-volume is cropped centered on the candidate box, and then resampled to an isotropic 1×1×1 mm. 3The HU value is normalized to the [0,1] interval. Then, it is input into a 3D U-Net++ network: the encoder extracts multi-scale features through 4 layers of downsampling (channel count 32→64→128→256→512), the decoder fuses multi-level features through nested dense skip connections, and finally outputs a foreign object probability map through a 1×1×1 convolution and a sigmoid function. The combined loss function during segmentation is 0.7×Dice Loss + 0.3×Boundary IoU Loss, where the predicted probability is... Figure 2 When obtaining the prediction mask through valueization, the threshold is set to 0.5. After the function training is completed, the inference output probability map is directly binarized (threshold 0.5), and then post-processed using a Conditional Random Field (CRF) (iteration 10 times) to smooth the boundaries, fill holes, and eliminate isolated noise, finally outputting the 3D mask of the foreign object. Through this process, the IoU of the foreign object boundary reaches over 0.86, and the distance error from the incisor is controlled within 2mm.

[0105] Specifically, in step S5, the distance from the incisors is calculated as the path length (in cm) along the esophageal centerline from the incisors to the projection point of the foreign body. This is obtained through calibration and mapping between the CT coordinate system and the endoscopic coordinate system. Specifically, this includes: automatically detecting the position of the incisors in CT images; reconstructing the esophageal centerline based on an esophageal segmentation mask; calculating the arc length along the centerline from the projection point of the foreign body to the incisors; and performing linear regression calibration on the endoscopic and CT measurements using pre-collected case data to obtain the mapping conversion formula. The final output is the path distance from the foreign body to the incisors, in centimeters, with an error controlled within 2 mm, which can be directly used for endoscopic surgical navigation. The formula for calculating the distance from the incisors is: ; The arc length (mm) of the projection of the center point of the foreign object onto the central line of the esophagus. The calibration coefficient (obtained through linear regression of multiple cases, typically ≈1.0) is used. The calibration constant is (mm).

[0106] The esophagus is automatically segmented based on anatomical landmarks, including the upper, middle, and lower segments. The upper segment is 15-23 cm from the incisors, extending from the esophageal inlet to the upper edge of the aortic arch. The middle segment is 23-32 cm from the incisors, extending from the upper edge of the aortic arch to the lower edge of the inferior pulmonary vein. The lower segment is 32-40 cm from the incisors, extending from the lower edge of the inferior pulmonary vein to the cardia. If there are significant differences in patient height, an individualized proportional segmentation method can be used, dividing the esophagus into upper, middle, and lower segments according to 0-30%, 30%-70%, and 70%-100% of the total esophageal length.

[0107] In one embodiment, the projected arc length =253mm, calibration coefficient =0.998; Calibration constant =2mm, calculated as follows =25.4cm, 23 < 25.4 < 32 cm, located in the middle of the esophagus.

[0108] Specifically, in step S5, based on the 3D point cloud of the foreign object, the distance of the maximum spatial diagonal is taken as the maximum diameter of the foreign object. The 3D point cloud originates from all voxels or boundary voxels in the 3D segmentation mask of the foreign object (output by the segmentation network in step S4). All point coordinates use physical coordinates (unit: mm), and voxel indices are converted to physical coordinates using pixel spacing and layer thickness in DICOM metadata. The maximum diameter of the foreign object refers to the maximum Euclidean distance between any two points in the 3D point cloud, requiring traversal of all point pairs and calculation of the distance between all point pair combinations. A naive traversal method can be used for calculation. For large point clouds, an optimized algorithm using convex hull + rotating caliper can be used to reduce computational complexity. First, the 3D convex hull of the point cloud is calculated, and then the maximum distance is calculated at the vertices of the convex hull. The overall computational complexity is reduced from O(N^2) to O(N^2). 2 The value is reduced to O(N log N + H). 2 The naive traversal method has a time complexity of O(N). 2 The first algorithm is suitable for scenarios with N ≤ 5000 (small foreign objects), while the second algorithm, convex hull + rotating caliper, is suitable for scenarios with N > 5000 (large foreign objects). An approximation method (randomly selected) can also be used, with a complexity of O(K). 2 K << N, suitable for scenarios with extremely high real-time requirements.

[0109] The distance between the diagonals in space is calculated using the three-dimensional Euclidean distance formula. ;in, For the first Physical coordinates of an individual element, in mm; For the first Physical coordinates of an individual element, in mm; Let be the Euclidean distance between the two points, in mm.

[0110] It should be noted that the voxels must be isotropic before calculation. Otherwise, Euclidean distance will introduce deformation errors. If the original voxels are anisotropic, they need to be resampled to 1×1×1 mm. 3 All point coordinates must be in the same physical coordinate system (usually the DICOM RAI coordinate system). Inconsistent coordinate systems will lead to distance calculation errors. The distance calculation result is in millimeters (mm). If centimeters (cm) are required, divide by 10 to convert. Surface point cloud computing is faster, but the endpoint of the maximum diameter may be inside (rare). Global point cloud is more accurate but computationally more expensive. Global point cloud is recommended.

[0111] In one embodiment, the foreign object mask comprises 2400 voxels with a physical size of 1.0 × 1.0 × 1.0 mm³ (isotropic), and a point cloud scale of N = 2400. The maximum diameter is calculated to be 20.1 mm. The calculation process is shown in Table 2.

[0112] Table 2

[0113]

[0114] The method for calculating the volume of foreign matter includes counting the total number of effective voxels and calibrating the volume of a single voxel, as detailed below:

[0115] Effective voxel count: Traverse the 3D mask M containing the foreign object and count the number of voxels marked as 1. Optionally, 26-neighborhood connected component analysis is used to remove isolated noise regions with volumes smaller than the minimum volume threshold, retaining only the largest connected component as an effective foreign object. In this embodiment, the minimum volume threshold is 10 voxels, and connected components smaller than this threshold are directly removed.

[0116] Individual voxel volume calibration: Read the pixel pitch (X-axis, e.g., 0.7 mm; Y-axis, e.g., 0.7 mm) and layer thickness (Z-axis, e.g., 1.25 mm) from DICOM metadata, and calculate the physical volume of an individual voxel. Unit: mm 3 If the original voxels are anisotropic (Δx ≠ Δz), trilinear interpolation resampling at 1×1×1 mm is recommended. 3 After isotropic voxels are used for unified calculation, at this time .

[0117] Foreign object volume calculation: Unit: mm 3 Optionally, the probability map P output by the segmentation network retains more information than the binary mask, and a probability-weighted volume can be used: Retaining some volume effect information improves the calculation accuracy of boundary voxels and reduces boundary errors. In one embodiment, the total number of voxels for a foreign object mask is N=2400, and the volume of a single voxel is 1 mm after isotropic resampling. 3 The volume of the foreign object is 2400 mm. 3 =2.4cm 3 .

[0118] CT value = mean ± standard deviation of HU values ​​of all voxels within the foreign body region; in the calculation of CT value, "within the region" refers to all voxels covered by the foreign body 3D segmentation mask, that is, the spatial region composed of all voxels marked as "foreign body" (value 1) in the foreign body mask output in step S4 and optimized by post-processing. The specific calculation method includes: (1) traversing all voxels with a value of 1 in the mask, extracting their corresponding HU values ​​in the original CT image, and obtaining a set; (2) using 3 Principle (Laida Criterion) for removing outliers: Calculate the mean and standard deviation Eliminate those that meet the requirements After recalculating the voxels, this step can be performed selectively. When this step is used, non-alien voxels introduced by segmentation error can be excluded; (3) Calculate the mean. and standard deviation : ; (4) Output , Unit HU.

[0119] In one embodiment, assuming the foreign object mask contains N = 9 voxels, the corresponding HU values ​​are shown in Table 3:

[0120] Table 3

[0121]

[0122] Detailed calculation steps:

[0123] Step 1, Summation: ;

[0124] Step 2: Calculate the mean: ;

[0125] Step 3: Calculate the squared deviation for each voxel; the results are shown in Table 4.

[0126] Table 4

[0127]

[0128] Step 4: Calculate the variance and standard deviation: ; ;

[0129] Step 5, Output Results: ;

[0130] That is, in this embodiment, the foreign object mask contains 2400 voxels, with a calculated average value. =450 HU, standard deviation =12.9 HU, output CT value = 450 ± 12.9 HU, indicating a bony foreign body (fish bone or bone fragment), with uniform density.

[0131] It should be noted that if the segmentation network outputs a probability map instead of a binary mask, a probability-weighted calculation can be used. This is to reduce some of the volumetric effect error.

[0132] It should be noted that a CT value > 1000 HU indicates a possible metallic foreign body, such as a coin or battery, which is clinically high-risk and prone to perforation; a CT value between 200-800 HU indicates a possible bony foreign body, such as a fishbone or bone fragment, which is clinically medium-risk and requires endoscopic removal; a CT value < 100 HU indicates a possible low-density foreign body, such as plastic or fruit pit, which is clinically low-risk and can be conservatively observed; a small standard deviation indicates a foreign body with uniform density and dense structure, making it less prone to breakage; a large standard deviation indicates a foreign body with uneven density, possibly bone fragments or a mixture of foreign bodies.

[0133] In step S5, the embedding risk assessment includes the extraction and fusion determination of three features:

[0134] Sharpness: Sharpness reflects the sharpness of protrusions on the surface of foreign objects. A higher value indicates a sharper foreign object and a greater risk of embedding / perforation. Surface point clouds are extracted from the 3D mask of the foreign object, and a local quadratic surface is fitted to each surface point to calculate the average curvature. A greater curvature indicates a sharper surface. The formula for calculating sharpness is: A higher sharpness value indicates a sharper foreign object. For the first The average curvature of a surface point; This is the curvature threshold, meaning that only significantly convex, sharp points are counted during the calculation. The total number of surface points; in one embodiment, the curvature threshold is set to 0.5, i.e., the sharpness calculation formula is: Total number of surface points =1200, total number of sharp points = 180, sum of average curvature of sharp points = 216, sharpness .

[0135] Irregularity: Irregularity reflects the degree to which the shape of a foreign object deviates from a spherical shape. A higher value indicates a more irregular shape and a greater likelihood of embedding. The calculation method is as follows: calculate the volume of the foreign object. and minimum circumsphere volume Irregularity ,like When the object is elongated or thin, it indicates that the object exceeds the minimum circumscribed sphere, so the convex hull volume ratio is used instead. A higher irregularity value indicates a more irregular shape. For example, consider a spherical foreign object; its volume... =500mm 3 , the maximum diameter =10mm, minimum circumscribed sphere volume =524mm 3 , / =0.95, irregularity =0.05. Taking a thin, long fishbone as an example, the volume of the foreign object is... =2400mm 3 , the maximum diameter =20 mm, minimum circumscribed sphere volume =4188mm 3 , / =0.57, irregularity =0.43. Taking an irregular bone fragment as an example, the foreign body volume =1800 mm 3 , the maximum diameter =18 mm, minimum circumscribed sphere volume =3054mm 3 , / =0.59, irregularity =0.41.

[0136] Scoring of abnormal areas on CT scans of surrounding tissues: This reflects the degree of damage to surrounding tissues caused by the foreign body (edema, inflammation, hemorrhage, emphysema, etc.) and is direct evidence of the risk of embedment. Morphological expansion of the foreign body mask (spherical structural elements, expansion) is performed. =5mm) After subtracting the original mask, the surrounding area mask is obtained. The abnormality type is determined based on the CT value range of the surrounding area. The voxel percentages of edema / inflammation (+50~+150 HU), hemorrhage (+50~+200 HU), and emphysema (-500~-200 HU) are calculated. Among them, the voxel percentage of edema / inflammation is The percentage of hemorrhage toxins was: The percentage of emphysema-related factors was: ;in, The total number of primes in the surrounding area; weighted summation of outlier scores. In one embodiment, the overall element of the surrounding area =8000; Edema / Inflammation voxels = 1200, percentage 0.15, weight 0.4, weighted value 0.060; Hemorrhage voxels = 400, percentage 0.05, weight 0.4, weighted value 0.020; Emphysema voxels = 0, percentage 0.00, weight 0.2, weighted value 0.000; then abnormal score =0.080.

[0137] The criteria for judging the three characteristics are as follows:

[0138] Sharpness: ≥0.6 indicates high risk, 0.30-0.60 indicates medium risk, and <0.3 indicates low risk.

[0139] Irregularity: ≥0.6 indicates high risk, 0.30-0.60 indicates medium risk, and <0.3 indicates low risk.

[0140] Anomaly score: ≥0.3 indicates high risk, 0.10-0.30 indicates medium risk; <0.1 indicates low risk.

[0141] Risk Level Determination: Three features are input into a pre-trained random forest model. A hard upgrade rule is applied, prioritizing the following rules: Rule 1 (highest priority): If any feature reaches a high-risk threshold (e.g., sharpness ≥ 0.6, irregularity ≥ 0.6, or anomaly score ≥ 0.3), it is directly classified as high-risk. Rule 2 (lower priority): The three features are input into the pre-trained random forest model, and the model outputs the probabilities of the three categories. The risk level corresponding to the highest probability is selected. Rule 3 (lowest priority): If the model output confidence level is ≥0.7, the model prediction result is used; otherwise, it reverts to the rule classification (two or more medium-risk features are judged as medium risk, otherwise as low risk). Through the above process, the objective quantitative classification of esophageal foreign body impaction risk is achieved. See Table 5 for examples of risk level determination.

[0142] Table 5

[0143]

[0144] In the method disclosed herein, step 1 involves preprocessing the original CT images to unify CT images acquired from different devices and with different scanning parameters into a standardized data format and grayscale range, eliminating inconsistencies caused by voxel anisotropy and differences in grayscale dynamic range. Specifically, this includes: DICOM reading to obtain the image matrix and metadata; trilinear interpolation resampling to 1mm×1mm×1mm isotropic voxels to avoid geometric parameter calculation distortion; adjusting the window width and level to a soft tissue window (350 HU window width, 40 HU window level) to highlight the contrast of the esophagus and foreign body; and normalizing the HU value to the [0,1] interval to meet the input distribution consistency requirements of deep learning models and accelerate model convergence. In step 2, a 3D CNN combined with an anatomical spatial attention mask is used for esophageal region localization. Compared to traditional 2D methods or manual delineation, this reduces the search space by 80% and preserves the depth features of the foreign body using three-dimensional spatial information. In step 3, FPN multi-scale feature extraction, SENet+CBAM dual attention enhancement, and an anchor-free detection head are used. Compared with existing anchor-based detection methods, anchor box optimization is avoided, and the detection sensitivity for small foreign objects such as fish bones is ≥90%, which is 15-20% higher than manual interpretation. In step 4, a 3D segmentation network combined with Canny edge detection for supervision and boundary loss function optimization is used. Compared with the segmentation method using only Dice Loss, the boundary error is significantly reduced, the distance error from the incisor is ≤2mm, and the boundary IoU is improved by 10-15%. In step 5, a precise quantification method based on 3D point cloud (maximum diameter, volume, CT value) and a random forest evaluation model based on multiple parameters (sharpness, irregularity, and surrounding tissue abnormalities) are used. Compared with clinical experience estimation, objective quantitative output of foreign object parameters and automated classification of embedding risk are achieved.

[0145] In some optional embodiments provided in this disclosure, the isotropic voxel resolution ranges from 0.5 mm × 0.5 mm × 0.5 mm to 2.0 mm × 2.0 mm × 2.0 mm. Preferably, it is 1.0 mm × 1.0 mm × 1.0 mm. Specifically: high resolution (0.5 mm) is suitable for scenarios requiring the detection of millimeter-sized foreign objects, but requires significant computation; standard resolution (1.0 mm) balances small foreign object detection rate with processing efficiency, suitable for most routine clinical applications; low resolution (above 1.5 mm) is only suitable for rapid screening or scenarios with limited computational resources, and the risk of missing extremely small foreign objects must be considered. The resolution in all three directions must be consistent (isotropic) to ensure the accuracy of subsequent 3D reconstruction and volume calculation.

[0146] In some optional embodiments provided in this disclosure, reference is made to Figure 5Step S2, based on a 3D convolutional neural network model and in conjunction with an anatomical spatial attention mask, detects the esophageal region in the preprocessed CT images and outputs the minimum 3D bounding box of the esophagus in the CT images, including the following steps: S21, based on the preprocessed CT images and the anatomical prior that the esophagus is located behind the trachea and in front of the spine, generates an anatomical spatial attention mask; S22, using a 3D convolutional neural network model, downsamples the preprocessed CT images layer by layer, extracts features, and obtains a bottom-level feature map; S23, fuses the anatomical spatial attention mask with the bottom-level feature map to obtain a fused bottom-level feature map; S24, using a 3D convolutional neural network model, upsamples the fused bottom-level feature map layer by layer to obtain an esophageal probability map; S25, after binarizing the esophageal probability map, selects the largest connected component, calculates the minimum bounding box of the largest connected component, and outputs the minimum 3D bounding box of the esophagus in the CT images.

[0147] Specifically, step S2 includes the following steps: S21, generating an anatomical spatial attention mask: generating a three-dimensional attention weight map based on anatomical priors (posterior to the trachea, anterior to the spine). It should be noted that the esophagus has a clear anatomical location, located posterior to the trachea, anterior to the spine, and to the left of the thoracic aorta (middle and lower segment), running within the mediastinum, extending from the pharynx (C6 level) to the gastroesophageal junction (T11 level). Therefore, the specific method includes: firstly, quickly locating the trachea, spine, and mediastinal boundaries in CT images; calculating the distance of each voxel to the posterior edge of the trachea, the anterior edge of the spine, and the center of the mediastinum; assigning high weight when the distance to the posterior edge of the trachea and the anterior edge of the spine are both within the distance threshold range and located within the mediastinal region; assigning medium weight when only the above two conditions are met; otherwise, assigning low weight; applying a three-dimensional Gaussian filter to the obtained initial weight map for smoothing, so that the attention mask transitions smoothly and avoids hard boundaries; finally generating an anatomical spatial attention mask; in one embodiment, step S21, generating an anatomical spatial attention mask... The force mask includes: based on the anatomical prior that the trachea is located anteriorly and the spine is located posteriorly, the trachea, spine, and mediastinal boundaries are first quickly located in the CT image. For each voxel, the distance to the posterior edge of the trachea, the anterior edge of the spine, and the center of the mediastinum is calculated. When the distance to the posterior edge of the trachea and the anterior edge of the spine is within the distance threshold range (3-10 mm in this embodiment) and is located within the mediastinal region, a high weight is assigned (1.0 in this embodiment); when only the above two conditions are met, a medium weight is assigned (0.5 in this embodiment); otherwise, a low weight is assigned. A three-dimensional Gaussian filter (in this embodiment) is applied to the obtained initial weight map. The area (2mm) is smoothed to ensure a smooth transition of the attention mask and avoid hard boundaries; finally, an anatomical space attention mask is generated.

[0148] S22, 3D U-Net encoder feature extraction: A multi-layer downsampling encoder is used, with each layer employing convolution + batch normalization + activation as the basic unit. Multi-scale feature maps from low to high dimensions are sampled and extracted layer by layer, with the size halved layer by layer and the number of channels doubled layer by layer, achieving size reduction and channel expansion. In one embodiment, a 4-layer downsampling encoder is used, with each layer containing two 3×3×3 convolutions, batch normalization, and LeakyReLU activation (negative slope = 0.01). Convolutional downsampling with a stride of 2 is used to extract multi-scale feature maps F0~F4 layer by layer, with the size halved layer by layer and the number of channels doubled layer by layer (32→512). The output feature maps are shown in Table 6 below.

[0149] Table 6

[0150]

[0151] In Table 1, Level 0 is the original resolution layer, and Levels 1-4 are downsampling layers.

[0152] S23. Fusion of Attention Mask and Feature Map: Because the lowest-level feature map has the strongest semantics and the lowest spatial resolution, it is most sensitive to the overall spatial layout. Therefore, fusion is performed on the lowest-level feature map. The specific operation process includes: inputting an anatomical spatial attention mask and the lowest-level feature map; downsampling the input anatomical spatial attention mask to the same size as the lowest-level feature map; broadcasting the downsampled attention mask to each channel of the lowest-level feature map; and then multiplying it voxel by voxel; thus enabling the network to focus on potential esophageal regions and suppress non-esophageal regions. In one embodiment, the fusion effect is shown in Table 7.

[0153] Table 7

[0154]

[0155] S24, U-Net Decoder and Skip Connections: A 4-layer upsampling decoder is used. Each layer includes transposed convolution upsampling (2x), splicing of skip connection features with the corresponding encoder layer, and two 3×3×3 convolutions. It fuses the features of each encoder layer, gradually restores the spatial resolution, and outputs a feature map of the same size as the input CT image.

[0156] S25. Output the esophageal probability map and 3D bounding box: The decoder outputs the esophageal probability map obtained through convolution and activation. After binarizing the probability map, the volume (voxels) of each connected region is calculated. The largest connected region is selected, and its minimum bounding box is calculated. The start and end coordinates of the esophageal 3D bounding box are then output. In one embodiment, the decoder output is obtained by 1×1×1 convolution and sigmoid activation. After binarization (the threshold is 0.5 in this embodiment), the volume (voxels) of each connected region is calculated. The connected region with the largest volume is retained as the esophageal region. The largest connected region is selected, and its minimum bounding box is calculated. The start and end coordinates of the esophageal 3D bounding box are then output, as shown in Table 8.

[0157] Table 8

[0158]

[0159] The final output is the smallest 3D bounding box of the esophagus, including the start coordinates, end coordinates, and dimensions. Through the above steps, automatic and precise localization of the esophageal region is achieved, reducing the search space by 80%.

[0160] In some optional embodiments provided in this disclosure, the three-dimensional convolutional neural network model is a 3D U-Net model or a 3D Mask R-CNN model.

[0161] In some optional embodiments provided in this disclosure, the Anchor-Free detection head is a CenterNet probe or a YOLOv8 probe.

[0162] In some optional embodiments provided in this disclosure, the main segmentation network is 3D U-Net++ or V-Net; the edge detection auxiliary supervision branch is the Canny edge detection branch.

[0163] In some optional embodiments provided in this disclosure, the evaluation model is a random forest model or a logistic regression model.

[0164] Specifically, this disclosure selects random forest or logistic regression as the embedded risk assessment model, mainly considering the following advantages: 1) The input features are only three-dimensional (sharpness, irregularity, and surrounding tissue abnormality score), which belongs to low-dimensional classification problems. Traditional machine learning models are more efficient than deep learning and are less prone to overfitting; 2) Strong interpretability. Random forest can output the ranking of feature importance, and logistic regression can display the coefficients of each feature, which is convenient for clinicians to understand the basis for risk judgment; 3) Fast training speed. Model iteration and verification can be completed quickly on the CPU; 4) Small model size. It is easy to embed into the PACS system to achieve lightweight deployment. When making a specific selection, if the focus is on accuracy and nonlinear interaction modeling, random forest is selected. If the focus is on interpretability and model stability, logistic regression is selected. The two can also be used in combination. The methods for obtaining the pre-trained model include: (1) Data collection and annotation: collect no less than 1200 annotated CT images of esophageal foreign bodies, including no less than 600 cases of fish bones, no less than 400 cases of bone fragments, and no less than 200 cases of metal foreign bodies. Two senior radiologists used ITK-SNAP software to annotate foreign body masks layer by layer, recording the type of foreign body and determining the embedding risk level based on clinical follow-up results (400 low-risk cases, 500 medium-risk cases, and 300 high-risk cases). To ensure the accuracy of data collection and annotation, inconsistencies in the initial annotation were arbitrated by other personnel. Consistency test: Kappa coefficient ≥ 0.85.

[0165] (2) Data augmentation: The training samples were augmented 5 times, including: random rotation (±10°) to simulate different body positions; random scaling (0.9-1.1 times) to simulate different body shapes; elastic deformation ( =2, =0.5), simulates breathing / swallowing movements; random flipping increases symmetry, simulates small foreign objects (5-50 voxels in volume), and alleviates imbalance in small samples. After enhancement, the total sample size reaches 6000 cases, effectively reducing the risk of overfitting.

[0166] (3) Feature extraction: Three features are extracted from each sample: sharpness (Based on surface curvature) Irregularity (Volume ratio), surrounding tissue abnormality score For example, Sample 1, the foreign object is a fishbone, with a medium risk level, a sharpness of 0.45, an irregularity of 0.38, and a surrounding tissue abnormality score of 0.08; Sample 2, the foreign object is a bone fragment, with a low risk level, a sharpness of 0.28, an irregularity of 0.42, and a surrounding tissue abnormality score of 0.31; Sample 3, the foreign object is metal, with a high risk level, a sharpness of 0.71, an irregularity of 0.58, and a surrounding tissue abnormality score of 0.31; Sample 4, the foreign object is a fishbone, with a high risk level, a sharpness of 0.62, an irregularity of 0.35, and a surrounding tissue abnormality score of 0.28; Sample 5, the foreign object is a bone fragment, with a medium risk level, a sharpness of 0.35, an irregularity of 0.55, and a surrounding tissue abnormality score of 0.12.

[0167] (4) Model training: A random forest classifier is used, with the following hyperparameters: number of trees n estimators =200, maximum depth max depth =7, minimum number of split samples min samples_split =5. Automatically balance class_weight=balanced. The model stability was evaluated using 5-fold cross-validation, with a test set accuracy of 86.1%, a macro-average F1 score of 85.4%, and a high-risk sensitivity of 90.0%.

[0168] (5) Dynamic adjustment of boundary loss (if end-to-end joint training is used): The boundary loss weights are fixed for the first 50 epochs. =0.3, no adjustment, first learn overall region segmentation; in rounds 51-150, if the IoU of the validation set boundary does not improve for 3 consecutive rounds, then... Increase by 0.05 (maximum 0.5) to force attention to boundary precision; in rounds 151-300, if boundary IoU ≥ 0.85, then... The error was gradually reduced to 0.2 to avoid overfitting boundary noise. Through dynamic adjustment, the boundary IoU was improved by about 7%, and the distance error from the incisor was reduced by 33%.

[0169] The final model file is saved as rf_esophageal_risk_model.pkl (approximately 2.5MB). The single-inference time is ≤1ms, and it can be embedded in a PACS system to achieve real-time risk assessment.

[0170] Reference Figure 2 In some optional embodiments provided in this disclosure, the method further includes: generating a foreign body detection heatmap using the Grad-CAM method, overlaying a three-dimensional mask of the foreign body onto the original CT image, and outputting a structured report.

[0171] In one embodiment, refer to Figure 4 This disclosure also provides an automatic detection and location system for esophageal foreign bodies, including:

[0172] The system includes an automatic esophageal region localization module 100, a small target foreign body enhanced detection module 200, a foreign body boundary fine segmentation module 300, and a clinical parameter calculation and risk assessment module 400; among which,

[0173] The esophageal region automatic localization module 100 is used to input the original CT image, preprocess the original CT image to obtain the preprocessed CT image, and based on the three-dimensional convolutional neural network model, in conjunction with the anatomical space attention mask, to detect the esophageal region in the preprocessed CT image and output the minimum three-dimensional bounding box of the esophagus in the CT image.

[0174] The small target foreign object enhancement detection module 200, coupled to the esophageal region automatic localization module 100, is used to input the minimum 3D bounding box. Through a feature pyramid network, multi-scale features are extracted from the CT image within the minimum 3D bounding box region through convolution operations. These features are then enhanced with channel attention and spatial attention to perform pixel-level detection of foreign objects within the esophageal region. The Anchor-Free detection head outputs candidate foreign object boxes and foreign object categories. The loss function during the detection process is: , ;in, To detect the total loss, For the focus of loss, For balance coefficient, To compare the losses;

[0175] The foreign object boundary fine segmentation module 300, coupled to the small target foreign object enhancement detection module 200, is used to input foreign object candidate boxes. For foreign objects within these candidate boxes, a 3D segmentation network is used for fine segmentation, outputting a foreign object probability map. The foreign object probability map is then post-processed to output a 3D mask of the foreign object. The post-processing includes a conditional random field algorithm or morphological closing operation. The 3D segmentation network includes a main segmentation network and an edge detection auxiliary supervision branch. The total loss function used in the segmentation process is: ;in, Total loss due to primary segmentation. To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2;

[0176] Total loss of primary partition The function is: ;in, For Dice's loss; For boundary loss; These are the Dice Loss weights, and their values ​​are the Dice loss weights. The boundary loss weight is set to 0.3.

[0177] Edge detection auxiliary loss The calculation uses the binary cross-entropy loss function;

[0178] The clinical parameter calculation and risk assessment module 400, coupled with the foreign body boundary fine segmentation module, is used to input a three-dimensional mask of the foreign body. Based on the three-dimensional mask of the foreign body, the module calculates the positional parameters, geometric parameters, and impaction risk. The positional parameters are the distance of the foreign body from the incisors and the segmented position of the foreign body in the esophagus. The geometric parameters include the maximum diameter of the foreign body, the volume of the foreign body, and the CT value of the foreign body. The impaction risk includes low-level risk, medium-level risk, and high-level risk.

[0179] In some optional embodiments provided in this disclosure, reference is made to Figure 6 It also includes an interpretability output module 500, which is coupled to the clinical parameter calculation and risk assessment module 400. The interpretability output module 500 is used to input the calculation and assessment results output by the clinical parameter calculation and risk assessment module and generate a foreign body detection heatmap using the Grad-CAM method.

[0180] Specifically, the interpretable output module is coupled to the clinical parameter calculation and risk assessment module. Input data includes: raw CT images (to provide anatomical background) and the 3D feature map of the last convolutional layer of the detection or segmentation module (for calculating gradient weights). A foreign body detection heatmap is generated using the Grad-CAM method. The steps are as follows: calculate the gradient of the feature map relative to the predicted score of the foreign body category; obtain the channel weights through global average pooling; perform weighted summation and ReLU activation; then scale and overlay the result onto the raw CT image, outputting it as a color heatmap. This heatmap visually demonstrates which areas the model prioritizes for foreign body detection, assisting doctors in understanding the AI's decision-making basis.

[0181] This disclosed automatic esophageal foreign body detection and localization system includes an automatic esophageal region localization module, which narrows the search space based on 3D CNN; a small target foreign body enhanced detection module, which combines multi-scale feature fusion and attention mechanisms; a foreign body boundary fine segmentation module, which improves segmentation accuracy through boundary loss optimization; and a clinical parameter calculation and risk assessment module, which outputs location, geometric parameters, and embedding risk. It is suitable for emergency and primary care hospital scenarios and has significant clinical application value.

[0182] The automatic detection and localization system for esophageal foreign bodies disclosed herein uses an NVIDIA RTX 3090 or a domestic AI accelerator card (such as Cambricon MLU370) as its computing platform; TensorRT or OpenVINO as its inference framework; and the entire system is configured with interfaces that support DICOM reception, PACS system integration, and automatic report writing.

[0183] The method disclosed herein provides a more precise foreign body mask with clearer boundaries. Overall, the entire process time of this embodiment is ≤30 seconds, the distance error from the incisors is ≤2mm, and the foreign body detection sensitivity is ≥90%. In contrast, traditional manual image reading methods result in blurred boundaries, rely on the doctor's subjective judgment, take approximately 2-3 minutes, have a distance error from the incisors of 5-8mm, and a foreign body detection sensitivity of 70-75%. This embodiment improves the processing time of a single CT image, enhances the distance error from the incisors and the foreign body detection sensitivity, and can generate structured reports that can be used for AI-assisted diagnosis, filling the technological gap in AI-assisted diagnosis of esophageal foreign bodies using CT images.

[0184] As can be seen from the above embodiments, the method and system for automatic detection of esophageal foreign bodies provided in this disclosure achieves at least the following beneficial effects:

[0185] 1. In the method disclosed herein, CT images of the esophagus region are input into a deep learning-based automatic detection system for esophageal foreign bodies using CT images for automatic detection and interpretation. This solves the problem of low efficiency and reliance on physician experience in CT interpretation of esophageal foreign bodies, and significantly improves the efficiency of emergency image interpretation. The processing time for a single CT image can be no more than 30 seconds.

[0186] 2. In the method disclosed herein, a three-dimensional convolutional neural network (3D CNN) is used to automatically locate the esophageal region, and a multi-scale feature fusion target detection algorithm is combined to achieve pixel-level detection and segmentation of foreign objects, solving the problem of easily missing small foreign objects; the foreign object detection sensitivity is not less than 90%, which is 15%-20% higher than manual interpretation;

[0187] 3. This method is based on CT images and optimization of a 3D segmentation network and boundary loss function. It can accurately segment detected foreign objects, calculate the location of the foreign object (upper / middle / lower esophagus, distance from incisors), geometric parameters (maximum diameter, volume, CT value), and assessment of the risk of embedment, thus improving the accuracy of 3D localization. It can achieve an error of no more than 2mm in the distance from incisors.

[0188] 4. In the system disclosed herein, the esophageal region automatic localization module, based on a three-dimensional convolutional neural network, can reduce the search space by 80%; the small target foreign body enhanced detection module, based on a multi-scale feature fusion network and attention mechanism, can achieve pixel-level detection of foreign bodies within the esophageal region; the foreign body boundary fine segmentation module can achieve fine segmentation of foreign bodies, improving localization accuracy; and the clinical parameter calculation and risk assessment module can obtain the precise location parameters, geometric parameters, and embedding risk of foreign bodies, which can assist in rapid manual decision-making, provide precise navigation for endoscopic examinations, and shorten operation time.

[0189] The improvements of this invention are not pieced together, nor do they increase the complexity of the overall structure or the complexity of use, nor do they significantly increase the cost. At the same time, this device is easy to install and disassemble, and is not easily damaged, which objectively improves the reliability of this invention. It meets the actual structural and application requirements of this field. For those skilled in the art, the improvements of this device are clear and complete, and it is easy to implement according to the contents of the specification, making it highly popular and adaptable.

[0190] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A method for automatic detection of esophageal foreign bodies, characterized in that, include: The original CT image of the esophagus region is acquired, and an original DICOM format image sequence is generated. The original DICOM format image sequence is preprocessed to obtain a preprocessed CT image. The preprocessing includes: transforming the anisotropic original voxel resampling into an isotropic voxel resolution; adjusting the window width and window level of the original CT image to a soft tissue window; and normalizing the HU value of the CT image to uniformly map it to the [0,1] interval. Based on a three-dimensional convolutional neural network model, and in conjunction with an anatomical spatial attention mask, the esophageal region is detected in the preprocessed CT image, and the minimum three-dimensional bounding box of the esophagus in the CT image is output. Within the minimum 3D bounding box region, a feature pyramid network is used to extract multi-scale features from the CT image within the region through convolution operations, resulting in a fused multi-scale feature map. This fused multi-scale feature map is then sequentially processed through channel attention enhancement and spatial attention enhancement to perform pixel-level foreign object detection within the esophageal region. Anchor-Free detection heads output candidate foreign object boxes and the foreign object category. Specifically, channel attention enhancement involves performing global average pooling, fully connected layers, and activation on the fused multi-scale feature map to generate a channel weight map. This channel weight map is then multiplied and fused with the fused multi-scale feature map channel-wise to obtain a channel-enhanced feature map. Spatial attention enhancement involves performing global max pooling and global average pooling on the channel-enhanced feature map, followed by convolution and sigmoid activation to generate a spatial weight map. This spatial weight map is then multiplied and fused with the channel-enhanced feature map position-wise to obtain a double-enhanced feature map. The foreign object categories include at least fish bones, bone fragments, and metallic foreign objects; the loss function during the detection process is: , ;in, To detect the total loss, For the focus of loss, For balance coefficient, To compare the losses; For the foreign objects in the candidate bounding box, a 3D segmentation network is used for fine segmentation, outputting a foreign object probability map. The foreign object probability map is then post-processed to output a 3D mask of the foreign objects. The post-processing includes a conditional random field algorithm or morphological closing operation. The 3D segmentation network includes a main segmentation network and an edge detection auxiliary supervision branch. The total loss function used in the segmentation process is... for: ;in, Total loss due to primary segmentation. To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2; The total loss of the main segment The function is: ; in, To divide the total loss, For Dice's loss; For boundary loss; These are the Dice Loss weights, and their values ​​are the Dice loss weights. The boundary loss weight is set to 0.

3. The edge detection auxiliary loss The calculation uses the binary cross-entropy loss function; Based on the 3D mask of the foreign body, clinical parameters are calculated, including location parameters, geometric parameters, and impaction risk. The location parameters are the distance of the foreign body from the incisors and the segmented position of the foreign body in the esophagus, obtained through calibration and mapping between the CT coordinate system and the endoscopic coordinate system. The geometric parameters include the maximum diameter of the foreign body, the volume of the foreign body, and the CT value of the foreign body. The maximum diameter of the foreign body is calculated by taking the distance of the maximum spatial diagonal based on the 3D point cloud of the foreign body. The volume of the foreign body is calculated by performing voxel-level segmentation on the foreign body target in the CT image, counting the number of effective voxels, obtaining the total number of effective voxels and the volume of a single voxel, and then using the total number of voxels... The volume of the foreign object is obtained by calculating the volume of a single voxel; the CT value refers to the mean ± standard deviation of the voxel HU values ​​within the region, and the calculation method of the CT value includes: calculating the arithmetic mean and standard deviation of the CT values ​​of all voxels within the foreign object mask; the embedding risk includes low-level risk, medium-level risk, and high-level risk, and the calculation method of the embedding risk includes: extracting the sharpness of the foreign object based on the curvature change of the mask surface of the three-dimensional mask of the foreign object; extracting the irregularity of the foreign object based on the volume ratio of the volume of the foreign object to the volume of the minimum circumscribed sphere of the foreign object; extracting the abnormal CT value region features of the tissue surrounding the foreign object; inputting the sharpness of the foreign object, the irregularity of the foreign object, and the abnormal CT value region features of the tissue surrounding the foreign object into a pre-trained evaluation model, and outputting the embedding risk level.

2. The method according to claim 1, characterized in that, The isotropic voxel resolution is 0.5mm×0.5mm×0.5mm-2.0mm×2.0mm×2.0mm.

3. The method according to claim 1, characterized in that, The process of detecting the esophagus region in the preprocessed CT image based on a three-dimensional convolutional neural network model, combined with an anatomical spatial attention mask, and outputting the minimum three-dimensional bounding box of the esophagus in the CT image includes the following steps: Based on the preprocessed CT images, and according to the anatomical prior that the esophagus is located behind the trachea and in front of the spine, an anatomical spatial attention mask is generated. A three-dimensional convolutional neural network model is used to downsample the preprocessed CT images layer by layer, extract features, and obtain the bottom feature map. The anatomical space attention mask is fused with the bottom feature map to obtain the fused bottom feature map; A three-dimensional convolutional neural network model is used to upsample the fused low-level feature map layer by layer to obtain the esophageal probability map. After binarizing the esophageal probability map, the largest connected component is selected, the minimum bounding box of the largest connected component is calculated, and the minimum three-dimensional bounding box of the esophagus in the CT image is output.

4. The method according to claim 1, characterized in that, The three-dimensional convolutional neural network model is either the 3D U-Net model or the 3D Mask R-CNN model.

5. The method according to claim 1, characterized in that, The Anchor-Free detector is either a CenterNet detector or a YOLOv8 detector.

6. The method according to claim 1, characterized in that, The main segmentation network is a 3D U-Net++ or V-Net; The edge detection auxiliary supervision branch is the Canny edge detection branch.

7. The method according to claim 1, characterized in that, The evaluation model is either a random forest model or a logistic regression model.

8. The method according to claim 1, characterized in that, The method further includes: A foreign body detection heatmap is generated using the Grad-CAM method. The three-dimensional mask of the foreign body is then superimposed on the original CT image to output a structured report.

9. An automatic detection and positioning system for esophageal foreign bodies, characterized in that, include: The system includes an automatic esophageal region localization module, a small target foreign body enhanced detection module, a foreign body boundary fine segmentation module, and a clinical parameter calculation and risk assessment module; among these, The esophageal region automatic localization module is used to input the original CT image, preprocess the original CT image to obtain the preprocessed CT image, and based on the three-dimensional convolutional neural network model, in conjunction with the anatomical space attention mask, to detect the esophageal region in the preprocessed CT image and output the minimum three-dimensional bounding box of the esophagus in the CT image. The small target foreign object enhancement detection module is coupled to the esophageal region automatic localization module. It is used to input the minimum 3D bounding box, and through a feature pyramid network, performs convolution operations on the CT image within the minimum 3D bounding box region to extract multi-scale features. These features are then sequentially enhanced through channel attention and spatial attention to perform pixel-level detection of foreign objects within the esophageal region. The Anchor-Free detection head outputs candidate foreign object boxes and foreign object categories. The loss function during the detection process... for: , ;in, To detect the total loss, For the focus of loss, For balance coefficient, To compare the losses; The foreign object boundary fine-grained segmentation module is coupled to the small target foreign object enhanced detection module. It is used to input the foreign object candidate box, perform fine segmentation of the foreign objects in the candidate box using a 3D segmentation network, output a foreign object probability map, and perform post-processing on the probability map to output a 3D mask of the foreign object. The post-processing includes a conditional random field algorithm or morphological closing operation. The 3D segmentation network includes a main segmentation network and an edge detection auxiliary supervision branch. The total loss function used in the segmentation process is... for: ;in, Total loss due to primary segmentation. To provide auxiliary loss for edge detection, The auxiliary loss weight for edge detection is set to 0.2; The total loss of the main segment The function is: ; in, To divide the total loss, For Dice's loss; For boundary loss; These are the Dice Loss weights, and their values ​​are the Dice loss weights. The boundary loss weight is set to 0.

3. The edge detection auxiliary loss The calculation uses the binary cross-entropy loss function; The clinical parameter calculation and risk assessment module is coupled to the foreign body boundary fine segmentation module. It is used to input the three-dimensional mask of the foreign body and calculate the position parameters, geometric parameters, and impaction risk based on the three-dimensional mask of the foreign body. The position parameters are the distance of the foreign body from the incisors and the segmented position of the foreign body in the esophagus. The geometric parameters include the maximum diameter of the foreign body, the volume of the foreign body, and the CT value of the foreign body. The impaction risk includes low-level risk, medium-level risk, and high-level risk.

10. The automatic esophageal foreign body detection and positioning system according to claim 9, characterized in that, It also includes an interpretability output module, which is coupled to the clinical parameter calculation and risk assessment module and is used to input the calculation and assessment results output by the clinical parameter calculation and risk assessment module, and generate a foreign body detection heatmap using the Grad-CAM method.

Images