Lightweight multi-modal early gastric cancer collaborative auxiliary diagnosis system for edge devices

Abstract

The application provides an edge device-oriented lightweight multi-modal early gastric cancer collaborative auxiliary diagnosis system, which comprises a lightweight multi-modal classification module, a decision-level fusion unit, a first feature extraction network and a second feature extraction network; the decision-level fusion unit is used for evidence theory fusion of first classification probability and second classification probability, and outputs a multi-modal classification result; a lightweight segmentation module is constructed based on an encoding-decoding structure, an encoder of the lightweight segmentation module comprises a plurality of stacked lightweight bottleneck modules, a feature enhancement module is arranged in a skip connection of the encoder, and the feature enhancement module is used for pixel-level lesion segmentation of a white light endoscopy image; a gating cooperation unit is connected with the lightweight multi-modal classification module and the lightweight segmentation module respectively, and is used for generating a gating signal according to the multi-modal classification result; when the multi-modal classification result is positive, the lightweight segmentation module is activated to perform a segmentation task; and when the multi-modal classification result is negative, a subsequent segmentation process is terminated.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and artificial intelligence technology, specifically relating to a lightweight, multimodal early gastric cancer auxiliary diagnostic system for edge device deployment. Background Technology

[0002] Gastric cancer is one of the most prevalent and deadliest malignant tumors worldwide. The five-year survival rate for early-stage gastric cancer can reach over 90%, while the survival rate for advanced-stage gastric cancer drops significantly. Therefore, timely and accurate screening and diagnosis of early-stage gastric cancer are crucial for improving patient prognosis and reducing mortality.

[0003] Currently, endoscopy is the preferred method for detecting and diagnosing early gastric cancer. White light endoscopy is the most common screening method in clinical practice, capable of revealing the original morphology and color of the gastric mucosa. However, its ability to characterize the microvascular structure and subtle morphology of the lesion area is limited, easily leading to missed or false detections of small or flat lesions. Narrow-band imaging, as an emerging virtual chromatic endoscopy technique, enhances the absorption effect of specific wavelengths of light through special filters, significantly improving the contrast of microvascular and glandular morphology on the mucosal surface, thereby helping to improve the sensitivity of lesion detection. However, the diagnostic effectiveness of narrow-band imaging is highly dependent on the operator's experience and equipment parameters; its sensitivity will be significantly reduced in the hands of inexperienced physicians.

[0004] Traditional manual endoscopic diagnostic methods are limited by subjective factors such as physician experience, fatigue, and attention, resulting in a high risk of missed diagnoses and misdiagnoses, and poor diagnostic consistency among physicians with different experience levels. To overcome these limitations, computer-aided diagnostic systems based on deep learning have emerged. Deep learning models can autonomously extract and learn multi-level pathological features from massive amounts of endoscopic images, achieving efficient and objective diagnostic analysis.

[0005] However, existing deep learning-assisted diagnostic methods still face many challenges. First, at the data modality level, most methods rely solely on endoscopic images from a single modality for training and inference, failing to effectively integrate complementary pathological information from multi-source heterogeneous data, such as the overall color features of white light images and the local microvascular features of narrowband images. Second, at the network architecture level, existing models are mostly based on general-purpose, computationally intensive deep neural networks. Their massive number of parameters and floating-point operations result in high inference latency and resource consumption, making local deployment and application difficult on edge devices with limited computing resources. This severely restricts the widespread adoption of medical AI technology in primary healthcare institutions and mobile diagnostic scenarios. Finally, in terms of diagnostic functionality, existing methods are mostly limited to single classification or segmentation tasks, lacking a lightweight solution that can organically coordinate automatic classification screening with fine lesion segmentation, while effectively introducing and integrating doctors' prior clinical knowledge (such as textual descriptions of lesion locations) to further improve diagnostic accuracy and human-machine collaboration efficiency. Summary of the Invention

[0006] To address the shortcomings and deficiencies of existing technologies, this invention provides a lightweight, multimodal, collaborative auxiliary diagnostic system for early gastric cancer targeting edge devices. This system employs a modality-specific feature extraction network to capture global color features from white light endoscopic images and local spatial detail features from narrowband endoscopic images. It then uses evidence theory to fuse multimodal classification probabilities at the decision level, ensuring robust classification results even when a single modality is missing. Furthermore, a gated collaborative unit dynamically controls the activation and termination of the lightweight segmentation module based on the classification results, avoiding the waste of computational resources caused by false positives. The encoder of the lightweight segmentation module consists of stacked lightweight bottleneck modules with channel attention enhancement. Feature enhancement modules are configured in skip connections, utilizing multi-scale asymmetric deep convolution to fuse staged prediction masks. Features are refined in multiple dimensions through a series of channel, spatial, and pixel attention mechanisms combined with attention gates. The decoder restores resolution using a combination of lossless upsampling and lightweight feature transformation. Furthermore, the system can integrate medical text annotations as semantic priors, guide lesion segmentation through cross-modal feature joint analysis, and adaptively select the optimal classification and segmentation model configuration based on the modal combination of the input data. This approach significantly reduces computational complexity and the number of parameters while maintaining high accuracy, making it suitable for deployment on edge devices and enabling efficient, accurate, and collaborative assisted diagnosis of early gastric cancer endoscopic images.

[0007] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0008] A lightweight multimodal collaborative auxiliary diagnostic system for early gastric cancer for edge devices includes a lightweight multimodal classification module, a lightweight segmentation module, and a gating collaboration unit.

[0009] The lightweight multimodal classification module includes a first feature extraction network, a second feature extraction network, and a decision-level fusion unit. The first feature extraction network is constructed based on the imaging characteristics of white light endoscopy, used to extract lesion features from white light endoscopic images and generate a first classification probability. White light endoscopic images can present the original morphology and color of the gastric mucosa; subtle color gradation differences in tissue are key to diagnosis. Therefore, the first feature extraction network is designed to have a strong ability to extract color features. The second feature extraction network is constructed based on the imaging characteristics of narrow-band endoscopy, used to extract lesion features from narrow-band endoscopic images and generate a second classification probability. Narrow-band endoscopic images highlight lesion boundaries and structures by enhancing the contrast of mucosal microvessels; therefore, the second feature extraction network is designed to be highly sensitive to spatial details. The decision-level fusion unit is used to fuse the first and second classification probabilities using evidence theory, outputting a multimodal classification result. Using decision-level fusion instead of feature-level fusion can effectively integrate complementary multimodal information while maintaining the independence of each modal model, and naturally possesses the ability to handle modality loss.

[0010] The lightweight segmentation module is built on an encoder-decoder structure. Its encoder contains multiple stacked lightweight bottleneck modules, and feature enhancement modules are incorporated in the skip connections for pixel-level lesion segmentation of the white light endoscopic image. The white light endoscopic image provides the overall morphology and location information of the lesions, making it the optimal medium for segmentation. Using lightweight bottleneck modules as the basic unit of the encoder significantly reduces computational complexity while maintaining feature extraction capabilities. Incorporating feature enhancement modules in the skip connections effectively fuses shallow detail features with deep semantic features, improving segmentation accuracy, particularly in recognizing blurred lesion boundaries.

[0011] The gated collaborative unit is connected to both the lightweight multimodal classification module and the lightweight segmentation module. It generates a gate signal based on the multimodal classification result. When the multimodal classification result is positive, the lightweight segmentation module is activated to perform the segmentation task; when the multimodal classification result is negative, the subsequent segmentation process is terminated. Through this gated collaborative mechanism, the system can first perform rapid screening of the input image, then perform the computationally intensive segmentation task only on images suspected of early gastric cancer. This not only effectively suppresses false positives in lesion-free images by single-stage segmentation models but also significantly reduces the overall computational load of the system and improves inference speed.

[0012] Furthermore, the decision-level fusion unit employs Dempster-Shafer evidence theory for probabilistic fusion. Dempster-Shafer evidence theory effectively handles uncertain information and is suitable for fusing multimodal diagnostic results. When a white light endoscopy image or a narrow-band endoscopy image is detected as missing, the decision-level fusion unit sets the classification probability corresponding to the missing modality to 1 and continues the fusion calculation. This design allows the system to flexibly handle modality loss in practical applications, improving the system's robustness and practicality.

[0013] Furthermore, the basic unit of the first feature extraction network is a WLI-Block, employing an inverted residual structure, and includes a Ghost Module and an efficient channel attention mechanism (ECA) for extracting color features from white light endoscopy images. The Ghost Module can generate more feature maps with lower computational cost, while the ECA enhances information interaction between channels, enabling the network to focus more on color features that are important for diagnosis. This structural design significantly reduces the number of parameters and computational cost while maintaining a strong ability to extract color features from white light images.

[0014] Furthermore, the basic unit of the second feature extraction network is an NBI-Block, employing an inverted residual structure and containing multiple parallel deep convolutional kernels with different receptive fields, used to extract spatial detail features from narrow-band endoscopic images. Multiple parallel deep convolutional kernels with different receptive fields can comprehensively capture multi-scale contextual spatial information of narrow-band images, enabling the network to more sensitively capture morphological abnormalities of microvessels and glands. This design is specifically optimized for the texture features of narrow-band images, improving the ability to identify early gastric cancer lesions.

[0015] Furthermore, the lightweight bottleneck module is a channel attention-enhanced Ghost Bottleneck module. Based on Ghost Bottleneck, an efficient channel attention mechanism (ECA) is inserted after each Ghost Module. While Ghost Bottleneck is computationally efficient, its use of depthwise convolution and simple channel concatenation can easily lead to information loss between feature groups. By inserting an efficient channel attention mechanism (ECA) after each Ghost Module, the internal feature grouping of the Ghost Module can be broken, greatly enhancing the network's ability to capture channel feature information while maintaining low computational cost.

[0016] Furthermore, the feature enhancement module includes a multi-scale feature fusion mapping unit and a feature optimization unit. The multi-scale feature fusion mapping unit uses multiple asymmetric depthwise convolutions of different scales to extract local contextual features, and then concatenates the extracted features with the corresponding staged prediction masks from the decoding stage. Asymmetric depthwise convolutions can achieve a receptive field comparable to standard convolutions at a lower computational cost, and the staged prediction masks can supplement the spatially fused features with overall contextual information. The feature optimization unit sequentially employs channel attention, spatial attention, and pixel attention mechanisms to refine the concatenated features in multiple dimensions, and uses the upsampled signal from the same-level decoder as a gating signal to filter the refined features through an attention gate mechanism. The triple attention mechanism can filter and enhance features from different dimensions, and the attention gate mechanism can suppress background noise, making the network more focused on the lesion region.

[0017] Furthermore, the decoder of the lightweight segmentation module includes multiple lossless upsampling modules. These lossless upsampling modules employ a PixelShuffle operation for upsampling, followed by connection to a Ghost Module and an efficient channel attention mechanism (ECA). Traditional upsampling methods, such as bilinear interpolation, are prone to information loss, while the PixelShuffle operation achieves upsampling by recombining pixels along the channel dimension, thus avoiding information loss. Combining the Ghost Module and the efficient channel attention mechanism enables high-quality spatial resolution reconstruction while maintaining lightweight characteristics.

[0018] Furthermore, the system also includes a text feature fusion module for receiving medical text annotations and extracting text feature vectors. The lightweight segmentation module is also used to concatenate the spatially aligned text feature vectors with the image features processed by the feature enhancement module, generating cross-modal joint features to guide lesion segmentation. In clinical practice, doctors often provide textual descriptions of lesion locations, which contain important prior clinical knowledge. By introducing medical text annotations as semantic guidance, the doctor's experience can be effectively utilized to further improve segmentation accuracy, especially for lesions with blurred boundaries or atypical morphologies.

[0019] Furthermore, the system also includes a modality adaptive selection unit, used to automatically select the corresponding combination of classification and segmentation networks based on the modality type of the input data. In actual clinical applications, different medical institutions may have different equipment, and the endoscopic image modalities they can provide are not entirely the same. The modality adaptive selection function enables the system to flexibly adapt to various input situations. When only white light images are available, the system calls the combination of a white light image classification network and a basic segmentation network; when white light, narrowband images, and medical text annotations are available simultaneously, the system automatically calls the combination of a fusion classification network and a text-guided segmentation network, achieving dynamic and efficient assisted diagnosis.

[0020] Furthermore, the multiple intermediate decoding layers of the lightweight segmentation module are configured to output staged prediction masks. These staged prediction masks, fused with the shallow encoded features of the corresponding stage, are then input to the feature enhancement module. The staged prediction masks can not only be used for deep supervised training, constraining the network's feature learning from shallow to deep layers, but also serve as input to the feature enhancement module, supplementing the spatial fusion features with overall contextual information. This helps the network better understand the overall morphology and location of lesions, thereby improving segmentation accuracy.

[0021] Compared with existing technologies, the present invention and its preferred embodiment, through a first feature extraction network and a second feature extraction network designed for the imaging characteristics of white light endoscopy and narrowband endoscopy respectively, can fully explore complementary pathological information of different modalities. The white light network focuses on color features, while the narrowband network focuses on microvascular and spatial detail features, effectively making up for the deficiency of insufficient information representation of a single modality.

[0022] Its decision-level fusion unit uses evidence theory to fuse multimodal classification probabilities and robustly handles missing modalities, avoiding system failure due to missing single-modal images and improving reliability and flexibility in clinical applications.

[0023] The gated collaboration unit uses the multimodal classification results as the gate signal and activates the lightweight segmentation module only when the classification result is positive. This suppresses the invalid computation caused by false positives, realizes efficient collaboration between classification and segmentation tasks, and significantly reduces the overall inference latency and power consumption of the system, making it particularly suitable for edge computing scenarios.

[0024] The encoder of the lightweight segmentation module uses a stack of channel attention-enhanced Ghost Bottleneck modules, which significantly reduces the number of parameters and floating-point operations while maintaining feature expressiveness. The feature enhancement module in the skip connections captures local context through multi-scale asymmetric deep convolution and uses staged prediction masks as supplementary information to improve the richness of feature fusion and spatial localization accuracy.

[0025] The feature optimization unit cascades channel attention, spatial attention, and pixel attention, and uses the upsampled signal from the decoder to perform feature filtering through an attention gate mechanism. This refines the features from multiple dimensions, significantly enhancing the feature response of the lesion region and improving the precision of the segmentation boundary.

[0026] The decoder uses a lossless upsampling module based on PixelShuffle, which avoids the information loss of traditional upsampling. Combined with subsequent lightweight feature transformation, it achieves a balance between high-quality resolution restoration and parameter efficiency.

[0027] By integrating semantic priors from medical text annotations, the system can guide the segmentation process with cross-modal joint features, organically combining physicians' clinical knowledge with image features, further improving the accuracy of segmentation and the effectiveness of human-machine collaborative diagnosis.

[0028] The modality adaptive selection unit enables the system to dynamically switch the optimal network combination based on the actual input modality type, eliminating the need to deploy a separate system for each modality combination, thus greatly improving the system's versatility and deployment efficiency.

[0029] Through a systematic lightweight design, the overall architecture significantly reduces model size and inference latency while maintaining detection and segmentation accuracy comparable to large models, enabling complex multimodal auxiliary diagnostic models to achieve real-time, localized inference on resource-constrained edge devices. Attached Figure Description

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0031] Figure 1 The diagram shows the structure of the lightweight feature extraction unit WLI-Block designed for white light images in this embodiment of the invention; in the diagram, (a) is the module structure with a step size of 1, and (b) is the module structure with a step size of 2.

[0032] Figure 2 The diagram shows the structure of the lightweight feature extraction unit NBI-Block designed for narrowband images in this embodiment of the invention; in the diagram, (a) is the module structure with a step size of 1, and (b) is the module structure with a step size of 2.

[0033] Figure 3 This is a structural diagram of WLI-Net, a lightweight white light modality classification network in an embodiment of the present invention.

[0034] Figure 4 This is a structural diagram of the Narrowband Modal Lightweight Classification Network NBI-Net in an embodiment of the present invention.

[0035] Figure 5This is an architecture diagram of FusionWN-Net, a lightweight multimodal classification model in an embodiment of the present invention.

[0036] Figure 6 This is a general framework diagram of the lightweight segmentation model GFDU-Net in an embodiment of the present invention.

[0037] Figure 7 The diagram shows the structure of the lightweight feature extraction module CAG-bneck in this embodiment of the invention; in the diagram, (a) is the module structure with a step size of 1, and (b) is the module structure with a step size of 2.

[0038] Figure 8 The figure shows the structure and technical details of the feature enhancement module FEM in the embodiment of the present invention; in the figure, (a) is the overall structure of FEM, (b) is the structure of the efficient channel attention mechanism ECA, (c) is the structure of the spatial attention mechanism SPA, (d) is the structure of the pixel attention mechanism PA, and (e) is the structure of the attention gate mechanism GA.

[0039] Figure 9 This is a schematic diagram of the decoding operation and the lossless upsampling module LUM in an embodiment of the present invention.

[0040] Figure 10 This is an overall architecture diagram of the lightweight multimodal segmentation model LVEU-Net in an embodiment of the present invention.

[0041] Figure 11 This is a structural diagram of the image-text feature fusion module in an embodiment of the present invention.

[0042] Figure 12 This is a flowchart of the lightweight multimodal collaborative diagnostic framework in an embodiment of the present invention.

[0043] Figure 13 This is an example diagram illustrating the effect of the collaborative diagnostic framework in suppressing false positive segmentation in an embodiment of the present invention. Detailed Implementation

[0044] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0045] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] This invention provides a lightweight, multimodal, collaborative, assisted diagnostic system for early gastric cancer deployed on edge devices. First, two structure-specific lightweight classification models are designed to efficiently extract lesion features from different modalities and perform single-modal classification, taking into account the different imaging characteristics of white light images and narrowband images. Then, a decision-level fusion algorithm is used to fuse the classification probabilities of the two modalities using evidence theory, integrating complementary multimodal information to obtain more accurate classification results. Simultaneously, this invention also constructs a lightweight lesion segmentation model and further incorporates medical text annotations from doctors as semantic guidance, achieving higher-precision segmentation through cross-modal feature fusion. Finally, the classification and segmentation models are integrated into a collaborative diagnostic framework, using the classification results as gating signals to dynamically control the execution of downstream segmentation tasks, effectively suppressing false positives and saving computational resources.

[0048] This invention achieves high-precision early gastric cancer diagnosis with extremely low computational complexity by designing a modality-specific lightweight feature extraction backbone, a multimodal classification based on decision-level fusion, and a "classification-segmentation" gating collaborative mechanism, combined with semantic-guided segmentation of medical text. Experiments show that this multimodal collaborative diagnostic framework achieves superior performance to existing technologies in lesion classification and segmentation tasks with only a small number of parameters and computational cost, and can be successfully deployed on mobile terminal devices.

[0049] The invention will now be described in further detail with reference to the accompanying drawings.

[0050] (I) Construction of a lightweight multimodal classification model

[0051] To efficiently distinguish between early gastric cancer and benign lesions, this invention first constructs a lightweight multimodal classification model based on heterogeneous neural networks. The design concept is to address the differences in imaging characteristics between white light and narrow-band endoscopic modalities by designing structurally specialized feature extraction units to fully mine and effectively fuse multimodal information with minimal computational resources.

[0052] 1. Modality-specific lightweight feature extraction backbone

[0053] WLI-Block Design for White Light Images: White light imaging exhibits a uniform and rich color gamut distribution, and subtle color gradation differences within tissues are crucial for diagnosis. Therefore, this invention designs a WLI-Block feature extraction unit, the structure of which is as follows: Figure 1 As shown, this module adopts an inverse residual structure of "coarse in the middle and fine at both ends." Specifically, it is implemented as follows: First, efficient channel dimensionality enhancement is performed through the Ghost Module, followed by an efficient channel attention mechanism (ECA) to break down feature grouping caused by pooling operations and enhance information interaction between channels. In the feature extraction part, another efficient channel attention mechanism (ECA) is introduced in parallel to fully learn the rich inter-channel color change information of the white light image. Simultaneously, feature reuse is achieved through Shortcut connections. This structure significantly reduces the number of parameters and computational cost while maintaining a powerful ability to extract color features from white light images.

[0054] NBI-Block Design for Narrow-Band Imaging: Narrow-band imaging enhances the contrast of mucosal microvessels to highlight lesion boundaries and structures, thus requiring the model to be highly sensitive to spatial details. To address this, this invention designs an NBI-Block feature extraction unit, the structure of which is as follows: Figure 2 As shown, this module also employs an inverted residual structure, but its feature extraction part consists of multiple parallel deep convolutional kernels with different receptive fields to comprehensively capture the multi-scale contextual spatial information of narrowband images. Simultaneously, to enhance feature reuse, a 1×1 convolutional kernel is introduced in the Shortcut connection for feature map channel matching. This design enables NBI-Block to more sensitively capture morphological abnormalities of microvessels and glands.

[0055] 2. Unimodal classification networks WLI-Net and NBI-Net

[0056] Based on the above basic units, this embodiment constructs lightweight classification networks for single-modality systems.

[0057] WLI-Net: such as Figure 3 As shown, WLI-Net consists of an initial module, multiple stacked WLI-Blocks, and a classifier. The initial module performs preliminary feature extraction and channel upscaling through a standard 3×3 convolution. The feature extraction backbone consists of alternating stacks of WLI-Blocks with stride 1 (extracting local details) and stride 2 (downsampling to construct high-level semantics). Finally, the classifier outputs the lesion probability distribution under white light modality through 3×3 convolution, global pooling, Dropout, and fully connected layers.

[0058] NBI-Net: e.g. Figure 4 As shown, the architecture of NBI-Net is similar to that of WLI-Net, but its feature extraction backbone consists of multiple NBI-Blocks to adapt to the extraction of texture features from narrowband images.

[0059] As a preferred implementation, in this embodiment, both the input white light image and the narrowband image are uniformly scaled to 224×224 pixels, with 3 channels. The initial module of WLI-Net uses a standard 3×3 convolution with 16 output channels; the feature extraction backbone consists of 6 stacked WLI-Blocks, with WLI-Blocks of stride 2 located at the 2nd, 4th, and 6th positions, and the number of output channels at each stage being 32, 64, and 128 respectively; the 3×3 convolution in the classifier part has 256 output channels, and the fully connected layer has an output dimension of 2. The overall architecture of NBI-Net is consistent with WLI-Net, but its feature extraction backbone consists of 6 NBI-Blocks, and the number of output channels at each stage is the same as that of WLI-Net.

[0060] 3. FusionWN-Net, a decision-level fusion model based on DS evidence theory

[0061] To integrate the complementary diagnostic advantages of single-modal models, this invention constructs a lightweight multimodal fusion classification model. For example... Figure 5 As shown, the aligned white light and narrowband images are input into the trained WLI-Net and NBI-Net respectively, and their prediction probability vectors P are obtained independently. WLI and P NBI Then, the Dempster-Shafer (DS) evidence theory is used for decision-level fusion, and the fusion rule is as follows:

[0062]

[0063]

[0064] in, This represents the predicted probability of the corresponding class under the WLI modality. This represents the predicted probability of the corresponding class under the NBI modality. This represents the final fusion probability for the corresponding category. Furthermore, when the algorithm encounters missing modes, it can overcome the potential mode loss problem in real-world applications by simply setting the predicted probability of the corresponding missing mode to 1.

[0065] This fusion method not only integrates independent judgments of different modalities, improving the accuracy of the final diagnosis (see Table 1 for comparison results), but also has a natural modality missing handling capability: when any modality data is missing, simply treating its predicted probability as 1 is enough to ensure the continuation of the diagnostic process.

[0066] Table 1 Performance comparison of the fusion classification model of this system with other advanced methods

[0067]

[0068] The experimental dataset used in this invention comes from different patients in six collaborating hospitals, including the Provincial Hospital Affiliated to Fuzhou University. It includes 7,020 white light endoscopy images and 5,337 narrow-band endoscopy images. The dataset was labeled by professional gastroenterologists. After alignment screening, 653 pairs of fully aligned multimodal endoscopy images were obtained as the test set. The remaining unaligned data were divided into training and validation sets in approximately a 7:2:1 ratio, ensuring that images from the same patient appeared in only one dataset.

[0069] (II) Construction of a lightweight multimodal segmentation model

[0070] 1. Lightweight single-modal segmentation model GFDU-Net

[0071] To achieve precise localization and delineation of lesion areas, this invention first designs a lightweight single-modal segmentation model, GFDU-Net, based on an encoder-decoder structure. Its overall framework is as follows: Figure 6 As shown.

[0072] Lightweight Feature Extraction Module CAG-bneck: To overcome the shortcomings of Ghost Bottleneck (G-bneck) which suffers from information loss between feature groups due to its use of depthwise convolution and simple channel concatenation, this invention proposes an efficient channel attention-enhanced Ghost Bottleneck, named CAG-bneck, with the following structure: Figure 7 As shown. CAG-bneck is based on G-bneck, and inserts an efficient channel attention mechanism ECA after each Ghost Module. The specific workflow of ECA is as follows: first, global average pooling is performed on the input features, and then an adaptive-size 1D convolution is used to capture the dependencies between channels. The kernel size k is determined by formula (3):

[0073]

[0074] Where C is the number of channels, γ=2, and b=1. Finally, the learned channel weights are normalized to the range of 0-1 using the Sigmoid function and multiplied with the original feature map to achieve adaptive recalibration of the channel feature responses. This design breaks the internal feature grouping of the Ghost Module, greatly enhancing the network's ability to capture channel feature information while maintaining low computational cost.

[0075] As a preferred implementation, the encoder part of GFDU-Net consists of eight stacked CAG-bnecks, with CAG-bnecks of stride 2 located at positions 2, 4, 6, and 8, and the number of output channels in each stage being 16, 32, 64, 128, and 256, respectively. The decoder part is set up with four decoding stages, each containing a lossless upsampling module (LUM) and a CAG-bneck, with the number of output channels in each stage being 128, 64, 32, and 16, respectively. Finally, the number of channels is mapped to 1 through 1×1 convolution, and a binary segmentation mask is output.

[0076] Feature Enhancement Module (FEM): To better uncover the boundary and spatial structure information of blurred lesions during segmentation, this invention designs a Feature Enhancement Module (FEM) in the skip connections of the network. For example... Figure 8 As shown, FEM consists of two consecutive steps: "multi-scale feature fusion mapping" and "feature optimization".

[0077] Multi-scale feature fusion mapping aims to capture and fuse shallow encoded features from different receptive fields. The process is as follows: First, the input shallow encoded feature x is processed using multiple asymmetric depthwise convolutions of different scales to obtain multiple local context features. .

[0078]

[0079]

[0080]

[0081] Next, the outputs of all branches are summed to obtain the aggregated features. :

[0082]

[0083] Finally, the aggregated feature is concatenated with the stage prediction mask P_{stage} generated in the corresponding decoding stage to obtain the spatial fusion feature. :

[0084]

[0085] in, These represent the shallow encoded features of the input. Indicates the current The first stage of the local feature extractor Feature maps generated by branch operations This represents the features obtained after passing through the local feature extractor. This represents the decoder prediction result at the corresponding stage. This indicates the final spatial fusion characteristics obtained. Indicates the convolution operation, subscript Indicates the kernel size, subscript This represents depthwise separable convolution. This indicates a splicing operation.

[0086] The feature optimization step aims to refine the spatial fusion features obtained in the previous step from different dimensions. It applies the following attention mechanisms in sequence: (1) using the efficient channel attention mechanism ECA to filter features from the channel dimension; (2) using the spatial attention mechanism SPA to focus on key spatial structural regions; and (3) using the pixel attention mechanism PA to capture fine-grained edge and texture information. The specific calculation process is as follows:

[0087]

[0088]

[0089]

[0090]

[0091] Finally, to suppress background noise, an attention gate mechanism is introduced. It uses the upsampled signal g output by the same-level decoder as the gate signal to fuse features. Filtering is performed to obtain the final enhanced features. :

[0092]

[0093] in, For channel attention mechanism, For spatial attention mechanisms, For pixel attention mechanism, For attention gate mechanism, This is an intermediate optimization result. It is the upsampled signal from the same level decoder, which is used as the gating signal input to the attention gate. This is the final output of the feature enhancement module.

[0094] Through orthogonal feature optimization of the FEM module, the model can more accurately focus on the edge contour and internal structure of the lesion.

[0095] Lossless Upsampling Module (LUM): In the decoding stage, this invention employs a lossless upsampling module (LUM) for spatial resolution reconstruction, such as... Figure 9As shown, LUM first employs a PixelShuffle operation to reorganize pixels along the channel dimension to avoid information loss during upsampling. Following this, a Ghost Module is used to match and integrate feature channels from the encoder, and an ECA module is added to break down feature groupings and enhance channel information interaction.

[0096] Deeply Supervised Training Strategy: During training, GFDU-Net introduces a deep supervision strategy, constraining the learning of intermediate layer features by generating prediction masks at each intermediate decoding layer and calculating the loss with the ground truth labels. Its total loss function is as follows:

[0097]

[0098]

[0099] in, and These represent Focal Loss and Dice Loss, respectively. These are the weights of the loss values ​​at different stages, as set in this embodiment. The values ​​are 1, 0.5, 0.4, 0.3, and 0.2. =0 to =4). This strategy can effectively constrain the network's feature learning from shallow to deep layers, thereby improving segmentation accuracy.

[0100] As a preferred implementation, all models in this invention are trained using the PyTorch deep learning framework with the AdamW optimizer. The initial learning rate is set to 0.001, and the weight decay coefficient is 1e-4. The training process employs a cosine annealing learning rate scheduling strategy, with a maximum of 150 training epochs and a batch size of 16. To prevent overfitting, Dropout layers are added after both fully connected and convolutional layers, with a dropout rate set to 0.2. Data augmentation strategies include random horizontal flipping, random vertical flipping, random rotation (-15° to +15°), random brightness adjustment (0.8-1.2x), and random contrast adjustment (0.8-1.2x).

[0101] 2. LVEU-Net, an image-text cross-modal fusion segmentation model

[0102] To further improve clinical adaptability, this invention proposes a lightweight multimodal segmentation model, LVEU-Net, based on GFDU-Net, utilizing textual descriptions of lesion locations commonly provided by doctors in real-world scenarios. Its architecture is as follows: Figure 10 As shown.

[0103] Text Feature Encoding: The system uses pre-trained BERT Embedding as the text feature encoder. For medical text annotations input by doctors (such as "the lesion is located on the anterior wall of the gastric antrum"), BERT Embedding converts them into text feature vectors containing contextual semantics through the summation and layer normalization operations of word segmentation, lexical embedding, position embedding and sentence segment embedding.

[0104] As a preferred implementation, this invention uses a pre-trained Chinese BERT-base model as the text feature encoder. This model contains 12 Transformer layers with a hidden layer dimension of 768. Medical text annotations adopt a unified structured format, such as "the lesion is located in [gastric antrum / gastric body / gastric fundus][anterior wall / posterior wall / greater curvature / lesser curvature]", which includes 24 standard description templates. During training, the BERT model is fine-tuned by freezing the parameters of the first 8 Transformer layers and only fine-tuning the last 4 layers and the output layer. The fine-tuning rounds are 20, and the learning rate is set to 2e-5. After one-dimensional convolution processing, the text features are compressed to 128 dimensions and then copied spatially to the same size (64×64) as the image feature map through a Tile operation.

[0105] Image-Text Feature Fusion Module: This module is the hub for achieving cross-modal fusion, such as... Figure 11 As shown, it employs a dual-branch processing mechanism. In the text modality branch, one-dimensional convolutions are sequentially applied to the text feature vectors output by BERT for local semantic enhancement. Then, the Tile operation is used to copy and stretch the vectors spatially, ensuring they have the same spatial size as the corresponding image feature maps, thus achieving spatial alignment. In the image modality branch, the shallow image features are first optimized through the Feature Enhancement Module (FEM) for multi-dimensional and multi-scale features, resulting in enhanced image features. Finally, the spatially aligned text features and the enhanced image features are concatenated along the channel dimension to generate image-text joint features that integrate cross-modal information, guiding the decoder to achieve more accurate lesion segmentation.

[0106] The comparative test results of the present invention and similar existing technologies are shown in Table 2. By introducing semantic prior, LVEU-Net achieves a significant improvement in segmentation performance compared to GFDU-Net.

[0107] Table 2 Performance Comparison of Different Segmentation Models

[0108]

[0109] (III) Lightweight Multimodal Collaborative Diagnostic Framework

[0110] To suppress false positives in lesion-free images using single-stage segmentation models and optimize computational efficiency, this invention proposes a "classification-segmentation" gated collaborative diagnostic framework, the process of which is as follows: Figure 12 As shown.

[0111] 1. Gated Collaboration Mechanism: When multimodal data is input, this framework first calls a lightweight multimodal classification model (such as FusionWN-Net) for preliminary screening. If the classification result is negative (not early-stage cancer), the subsequent segmentation task is terminated directly to avoid unnecessary computational resource consumption. Only when the classification result is positive (suspected early-stage cancer) is a gating signal generated to dynamically activate the downstream lightweight segmentation model (such as GFDU-Net or LVEU-Net) to perform precise lesion segmentation and localization.

[0112] 2. Modality Adaptive Selection: This framework can flexibly call different diagnostic model branches based on the modal differences of the actual input data. For example, when only white light images are available, the "WLI-Net+GFDU-Net" combination can be called; when white light, narrowband images, and medical text annotations are available simultaneously, the "FusionWN-Net+LVEU-Net" combination is automatically selected to achieve dynamic and efficient assisted diagnosis.

[0113] By combining classification and segmentation tasks, this framework achieves optimal segmentation accuracy while effectively suppressing false positives. A comparison with existing similar frameworks is shown in Table 3.

[0114] Table 3. Segmentation performance evaluation of the collaborative diagnostic framework

[0115]

[0116] As a direct demonstration of the effect, Figure 13 Examples of how this collaborative framework effectively suppresses false-positive segmentation are presented. When faced with non-cancerous images containing features easily misdiagnosed such as gastric mucosal bleeding and microvascular dilation, single-stage segmentation models generate a large number of false-positive segmentation regions. However, this collaborative framework, through gating filtering by a pre-classification model, accurately terminates the segmentation process without generating any mis-segments.

[0117] (iv) System development and edge device deployment

[0118] Based on the above models, the inventors' team also developed a lightweight multimodal assisted diagnostic system for PCs and a classification application for mobile devices. Mobile deployment tests show that WLI-Net and NBI-Net achieve frames per second (FPS) of 37.04 and 24.39 respectively on smartphones, approaching or reaching real-time diagnostic standards, while consuming only 209.4MB and 214.5MB of memory, fully demonstrating the high availability of the proposed method on resource-constrained edge devices.

[0119] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0121] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of lightweight, multimodal, early gastric cancer collaborative auxiliary diagnostic systems for edge devices. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A lightweight, multimodal, collaborative auxiliary diagnostic system for early gastric cancer targeting edge devices, characterized in that, include: A lightweight multimodal classification module includes a first feature extraction network, a second feature extraction network, and a decision-level fusion unit. The first feature extraction network is constructed based on the imaging characteristics of white light endoscopy and is used to extract lesion features from white light endoscopy images and generate a first classification probability. The second feature extraction network is constructed based on the imaging characteristics of narrowband endoscopy and is used to extract lesion features from narrowband endoscopy images and generate a second classification probability. The decision-level fusion unit is used to perform evidence theory fusion on the first classification probability and the second classification probability to output a multimodal classification result; The lightweight segmentation module is built on an encoder-decoder structure. Its encoder contains multiple stacked lightweight bottleneck modules, and its skip connections are equipped with feature enhancement modules for pixel-level lesion segmentation of the white light endoscope image. The gating coordination unit is connected to the lightweight multimodal classification module and the lightweight segmentation module respectively. It is used to generate a gating signal based on the multimodal classification result. When the multimodal classification result is positive, the lightweight segmentation module is activated to perform the segmentation task. When the multimodal classification result is negative, the subsequent segmentation process is terminated.

2. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The decision-level fusion unit uses Dempster-Shafer evidence theory for probabilistic fusion; when a white light endoscope image or a narrow band endoscope image is detected to be missing, the decision-level fusion unit sets the classification probability corresponding to the missing modality to 1 and continues to perform fusion calculation.

3. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The basic unit of the first feature extraction network is WLI-Block, which adopts an inverted residual structure and includes GhostModule and efficient channel attention mechanism ECA, used to extract color features from white light endoscope images.

4. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The basic unit of the second feature extraction network is NBI-Block, which adopts an inverted residual structure and contains multiple parallel deep convolutional kernels with different receptive fields, used to extract spatial detail features of narrowband endoscope images.

5. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The lightweight bottleneck module is a channel attention-enhanced Ghost Bottleneck module, which, based on Ghost Bottleneck, inserts an efficient channel attention mechanism (ECA) after each Ghost Module.

6. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The feature enhancement module includes a multi-scale feature fusion mapping unit and a feature optimization unit. The multi-scale feature fusion mapping unit uses multiple asymmetric depth convolutions of different scales to extract local context features and concatenates the extracted features with the stage prediction mask of the corresponding decoding stage. The feature optimization unit sequentially uses channel attention mechanism, spatial attention mechanism and pixel attention mechanism to refine the concatenated features in multiple dimensions, and uses the upsampled signal of the same level decoder as the gating signal to filter the refined features through the attention gate mechanism.

7. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The lightweight segmentation module's decoder includes multiple lossless upsampling modules, which use a PixelShuffle operation for upsampling, and are then connected to a Ghost Module and an efficient channel attention mechanism (ECA).

8. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The system also includes a text feature fusion module for receiving medical text annotations and extracting text feature vectors; the lightweight segmentation module is also used to perform channel concatenation of the spatially aligned text feature vectors with the image features processed by the feature enhancement module to generate cross-modal joint features to guide lesion segmentation.

9. The lightweight multimodal early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The system also includes a modality adaptive selection unit, which automatically selects the corresponding combination of classification network and segmentation network based on the modality type of the input data.

10. The lightweight, multimodal, early gastric cancer collaborative auxiliary diagnostic system for edge devices according to claim 1, characterized in that: The multiple intermediate decoding layers of the lightweight segmentation module are configured to output stage prediction masks, which are then fused with the shallow coding features of the corresponding stage and input to the feature enhancement module.

Images