A thyroid papillary carcinoma cell multi-modal detection system and method based on YOLOv11

Abstract

This application provides a YOLOv11-based multimodal detection system and method for papillary thyroid carcinoma cells, relating to the field of image detection technology. It addresses the problem in existing technologies of achieving both high accuracy and robustness in papillary thyroid carcinoma cell detection under limited computing power. The system specifically includes: an improved backbone network that extracts first-modal features from images at different scales by introducing C3k2_Faster and C2MSLA modules; a priori pyramid module for STEM processing and progressive downsampling of the image, and outputs second-modal features aligned with the first-modal features at different scales through convolutional projection; a gated fusion module for channel-level weighted fusion of the first and second-modal features at corresponding scales, outputting fused features; a neck module for transferring and aggregating the fused features; and a head module for outputting the target detection results. This application is used for detecting papillary thyroid carcinoma cells.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to a multimodal detection system and method for papillary thyroid carcinoma cells based on YOLOv11. Background Technology

[0002] Early and accurate cytological detection of papillary thyroid carcinoma is crucial for clinical diagnosis and treatment decisions. Currently, clinical practice mainly relies on manual interpretation of fine-needle aspiration (FNA) paraffin-embedded and stained slides. Some methods utilize deep learning technology, but most are based on single-modal images for detection. A few multimodal methods employ complex fusion structures. Manual interpretation suffers from high workload, strong subjectivity, and poor consistency among different interpreters, making it prone to missed detections and misjudgments in samples with low cell counts, high background impurities, or blurred morphological boundaries. Single-modal detection is susceptible to factors such as uneven staining depth, cell overlap, and background noise, resulting in insufficient detection stability. Existing multimodal methods, on the other hand, suffer from complex fusion structures, high computational costs, and difficulties in end-side deployment, making it difficult to balance detection accuracy, robustness, and inference efficiency. Therefore, current technologies struggle to achieve both high accuracy and high robustness in the detection of papillary thyroid carcinoma under limited computing power. Summary of the Invention

[0003] This application provides a YOLOv11-based multimodal detection system and method for papillary thyroid carcinoma cells, which solves the problem.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a YOLOv11-based multimodal detection system for papillary thyroid carcinoma cells is provided, comprising: an improved backbone network, a prior pyramid module, a gated fusion module, a neck module, and a head module. The improved backbone network, by introducing a C3k2_Faster module and integrating a C2MSLA module at the high semantic layer, is used to extract and output first-modal features of the image at three scales: P3, P4, and P5. The prior pyramid module performs STEM processing and progressive downsampling on the image, and outputs second-modal features aligned with the first-modal features at the P3, P4, and P5 scales through convolutional projection. The gated fusion module performs channel-level adaptive weighted fusion of the first and second-modal features at the corresponding scales, outputting fused features. The neck module is used for transferring and aggregating the fused features. The head module outputs bounding boxes and prediction confidence based on the aggregated and optimized fused features, obtaining the target detection result.

[0005] In conjunction with the first aspect mentioned above, in one possible implementation, the C3k2_Faster module includes: a first branch and a second branch; the first branch is used to preserve features of a portion of the input features after convolution decomposition; the second branch includes a cascaded Faster_Block, used to perform lightweight feature transformation and enhancement on another portion of the input features after convolution decomposition; the Faster_Block includes a partial convolutional spatial hybrid operator and a convolutional MLP and adopts a residual update mechanism.

[0006] In conjunction with the first aspect mentioned above, in one possible implementation, the C2MSLA module includes: a third branch and a fourth branch; the third branch is used to preserve the input features; the fourth branch includes a cascaded MSLAlock block for multi-scale texture extraction and context modeling enhancement of the input features.

[0007] In conjunction with the first aspect mentioned above, in one possible implementation, the gated fusion module includes: a GAP unit, a Concat unit, an MLP unit, a Sigmoid activation unit, and a fusion unit; the GAP unit is used to perform global average pooling on the first modality features and the second modality features at the same scale; the Concat unit is used to concatenate the pooled first modality features and the second modality features in the channel dimension to generate concatenated features; the MLP unit includes two layers of 1×1 convolution and a ReLU activation function, used to perform feature transformation on the concatenated features; the Sigmoid activation unit is used to output gate weights based on the concatenated vector after feature transformation through the Sigmoid activation function; the fusion unit is used to fuse the gate weights with the first modality features and the second modality features to obtain fused features.

[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the MSLAlock block includes: an MSLA unit and an FFN unit; the MSLA unit is used to divide the channel into four equal parts and perform 3×3, 5×5, 7×7, and 9×9 depth convolutions respectively, and to fuse and enhance the convolution results at each scale through a linear attention mechanism, while introducing learnable scale weights to adaptively adjust the contribution of features at different scales; the FFN unit is used to perform lightweight nonlinear transformation on the multi-scale fused features output by the MSLA unit.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the neck module uses an FPN structure to perform top-down feature transfer of fused features and a PAN structure to perform bottom-up feature aggregation of features.

[0010] Secondly, a YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells is provided, comprising: acquiring a first image and a second image; the first image is an outline image of an FNA sample after paraffin embedding and staining; the second image is a three-channel terahertz near-field internal detail image of the same visual region; performing multi-scale feature extraction on the first image based on an improved backbone network to obtain first modal features; performing STEM processing and progressive downsampling on the second image based on a prior pyramid module to obtain second modal features; performing channel-level adaptive weighted fusion of the first and second modal features through a gated fusion module to output fused features; performing aggregation optimization on the fused features based on a neck module; and outputting the target bounding box and prediction confidence based on the aggregated and optimized fused features through a head module to obtain the target detection result.

[0011] In conjunction with the second aspect mentioned above, in one possible implementation, the second image is processed using stem and progressive downsampling based on the prior pyramid module to obtain the second modality features. This includes: using the second modality input tensor corresponding to the second image as input, and obtaining an initial feature map through stem processing; performing progressive downsampling on the initial feature map sequentially to obtain three-scale intermediate feature maps; and performing scale adaptation and channel adjustment on the three-scale intermediate feature maps respectively through a 1×1 convolutional projection layer to output the second modality features aligned with scales P3, P4, and P5.

[0012] In conjunction with the second aspect mentioned above, in one possible implementation, a gated fusion module performs channel-level adaptive weighted fusion of the first modality features and the second modality features to output fused features. This includes: performing global average pooling on the first modality features and the second modality features at the same scale; concatenating the pooled first modality features and the second modality features in the channel dimension to generate concatenated features; performing feature transformation on the concatenated features and outputting gate weights through a sigmoid activation function; and fusing the gate weights with the first modality features and the second modality features to obtain fused features.

[0013] In conjunction with the second aspect mentioned above, in one possible implementation, the gating weights are fused with the first modality features and the second modality features to obtain fused features that satisfy the following formula:

[0014] in, For the fusion features at scale k, The gating weights are at scale k. This represents the first modal feature at scale k. This represents the second modal feature at scale k. For channel-by-channel multiplication, k is the scale index. .

[0015] This application provides a YOLOv11-based multimodal detection system and method for papillary thyroid carcinoma cells. It reduces computational overhead by introducing a C3k2_Faster module into an improved backbone network, while integrating a C2MSLA module at the high semantic layer to enhance multi-scale texture and context modeling capabilities, balancing model lightweighting and feature representation capabilities when acquiring first-modal features. A prior pyramid module performs stem processing, progressive downsampling, and convolutional projection on the image to achieve precise alignment with first-modal features at P3 / P4 / P5 scales, providing a foundation for cross-modal fusion. A gated fusion module performs channel-level adaptive weighted fusion at corresponding scales, dynamically allocating contribution weights between the two modalities to improve robustness in complex scenarios. The FPN / PAN structure aggregation of the neck module optimizes fused features, enhancing the detection capability for cells and clusters of different sizes, achieving high accuracy, high robustness, and efficient inference for papillary thyroid carcinoma cell detection under limited computing power.

[0016] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0017] Figure 1 A system architecture diagram of a YOLOv11-based multimodal detection system for papillary thyroid carcinoma cells is provided for embodiments of this application. Figure 2 A module architecture diagram of the C3k2_Faster module provided in this application embodiment; Figure 3 An architecture diagram of a Faster_Block provided for embodiments of this application; Figure 4 A module architecture diagram of a C2MSLA module provided in this application embodiment; Figure 5 An architecture diagram of an MSLAlock block provided in this application embodiment; Figure 6An architecture diagram of an MSLA unit provided in this application embodiment; Figure 7 A module architecture diagram of a gating fusion module provided in an embodiment of this application; Figure 8 A schematic flowchart illustrating a YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells provided in this application embodiment; Figure 9 A flowchart illustrating another YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells provided in this application embodiment; Figure 10 A flowchart illustrating another YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells provided in this application embodiment; Figure 11 This is a schematic flowchart of another YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells provided in this application embodiment. Detailed Implementation

[0018] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0019] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0020] The YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells provided in this application can be applied to, for example... Figure 1 The YOLOv11-based multimodal detection system for papillary thyroid carcinoma cells shown below, such as Figure 1 As shown, the system includes: an improved backbone network, a priori pyramid module, a gating fusion module, a neck module, and a head module, specifically: The improved backbone network replaces some feature extraction modules C3k2 in the original YOLOv11 backbone network with C3k2_Faster modules, and integrates C2MSLA modules in the high semantic layer to enhance multi-scale texture and context modeling capabilities. This enables the accurate extraction of first-modal features at three scales: P3 (1 / 8 downsampling), P4 (1 / 16 downsampling), and P5 (1 / 32 downsampling) from the first image. Among them, the C3k2_Faster module is as follows: Figure 2 As shown, by splitting the input features into two branches, one for feature preservation and the other for lightweight enhancement, and finally fusing the output, the core information of the original features is not lost, while the lightweight structure improves the efficiency of feature expression, adapting to the needs of edge deployment and efficient inference. Specifically, it includes a first branch (C3k=True) and a second branch (C3k=False). The first branch includes consecutively connected C3k blocks, used to preserve the core information of the split input features, avoiding feature loss due to excessive transformation. The second branch includes consecutively connected Faster_Block blocks, used to perform lightweight feature transformation and enhancement on another part of the input features after convolution decomposition, capturing fine-grained semantic information while reducing computational overhead and improving feature expression capabilities; Faster_Block is as follows... Figure 3 As shown, it includes partial convolutional spatial hybrid operators PConv and convolutional MLPs (including Conv and Conv2d) to replace traditional large-size convolutions. A random deactivation mechanism is introduced through the DropPath layer to suppress overfitting and improve model generalization ability. Residual connections are used to add the Faster_Block input and processed features element-wise to achieve smooth gradient propagation, alleviating the gradient vanishing problem in deep network training while preserving original feature information. The C2MSLA module is a multi-scale feature enhancement module based on the C2PSA structure, such as... Figure 4 As shown, the algorithm includes a third branch and a fourth branch. The third branch, after being split by the Split operator, directly retains some of the core semantic information of the input features, avoiding the loss of key features due to excessive transformation. The fourth branch, after being split by the Split operator, receives another portion of the input features, concatenates them with an MSLAlock block, and achieves progressive multi-scale feature enhancement. This involves multi-scale texture extraction and context modeling of the input features, strengthening the representation of complex features such as fine-grained cellular structures and cluster boundaries. Finally, the outputs of the third and fourth branches are concatenated to output the C2MSLA module. The MSLAlock block is shown below. Figure 5 As shown, it is composed of MSLA units and FFN units connected in series. The MSLA unit is as follows: Figure 6As shown, the input features are first divided into four equal parts along the channel dimension to form four sub-feature maps with the same number of channels. Parallel multi-scale feature extraction is performed through 3×3, 5×5, 7×7, and 9×9 depth convolutions, respectively. The 3×3 convolution focuses on fine-grained details such as cell nucleus-cytoplasm ratio and nuclear grooves. The 5×5 and 7×7 convolutions balance local cell structure and regional association information, and the 9×9 convolution captures the macroscopic outline of cell clusters. The four convolution results are first used to generate Query (Q), Key (K), and Value (V) vectors through learnable linear projection, respectively. Then, they are globally associated and fused through a linear attention mechanism. The complexity is reduced by reconstructing the attention calculation order. At the same time, learnable scale weight parameters are introduced to adaptively adjust the contribution of features of different scales in the fusion process (such as enhancing the 7×7 / 9×9 convolution feature weights for samples with blurred boundaries and strengthening the 3×3 convolution feature ratio for clear samples). The FFN unit is used to perform lightweight nonlinear transformation on the multi-scale fused features output by the MSLA unit.

[0021] The prior pyramid module is based on the input tensor corresponding to the second image. It first performs stem processing on the second image to obtain an initial feature map, then performs progressive downsampling through Conv2d, and performs scale adaptation and channel adjustment through a convolutional projection layer to generate a second modality feature that is strictly aligned with the first modality feature at the P3, P4, and P5 scales.

[0022] Gating fusion module such as Figure 7As shown, the system includes a GAP unit, a Concat unit, an MLP unit, a Sigmoid activation unit, and a fusion unit. These units are sequentially connected according to the feature processing flow: First, the GAP unit, as a feature preprocessing component, performs global average pooling on the first and second modal features at the same scale, compressing the two-dimensional feature maps of the two types of features into one-dimensional channel vectors. This preserves the semantic information of each channel while filtering out spatially redundant noise. The Concat unit receives the output from the GAP unit and concatenates the one-dimensional pooled vectors of the two modalities along the channel dimension, generating a concatenated feature that integrates the global channel correlation information of the two modalities. The MLP unit takes the concatenated feature as input and internally consists of two 1×1 convolutional layers and R... The system consists of an eLU activation function, a first-layer 1×1 convolution to compress the channel dimension and reduce computational overhead, a ReLU activation function to introduce non-linear feature transformation, a second-layer 1×1 convolution to restore the channel dimension and complete the accurate encoding of the bimodal correlation features, a Sigmoid activation unit to receive the concatenated vector encoded by the MLP unit, and to map the vector values ​​to the [0,1] interval through the Sigmoid activation function, outputting the gate weights used to dynamically characterize the contribution priority of the two modalities in each channel, the fusion unit takes the gate weights, the first modal feature and the second modal feature as input, and combines the gate weights with the two modal features respectively through channel-by-channel multiplication, and then adds the results to finally output the fused feature that integrates the complementary information of the two modalities.

[0023] The neck module employs a combined FPN and PAN structure, optimizing the fusion features at three scales: P3, P4, and P5. The FPN structure enables top-down feature transfer, upsampling the high semantic features of P5 and fusing them channel-by-channel with the features of P4 and P3 to supplement the semantic information of small-sized cell targets. The PAN structure facilitates bottom-up feature aggregation, downsampling the high-resolution features of P3 and feeding them back to the features of P4 and P5 to enhance the detailed localization of large-sized cell clusters. Finally, through bidirectional feature interaction, the deep optimization of multi-scale fusion features is achieved, providing feature support with both semantic integrity and rich detail for subsequent detection.

[0024] The head module takes the aggregated and optimized fused features output by the neck module as input and completes the target detection task through the detection head. The detection head has built-in classification and regression branches. The classification branch outputs the confidence score of the target as thyroid papillary carcinoma cells through lightweight convolution and activation functions. The regression branch accurately predicts the coordinates and size of the target box. After filtering redundant predicted boxes through non-maximum suppression (NMS), the final output includes the detection result containing the target box position and the corresponding confidence score.

[0025] To address the challenge of achieving both high accuracy and robustness in papillary thyroid carcinoma (PTC) cancer detection under limited computing power, this application provides a YOLOv11-based multimodal detection method for PTC cancer cells. This method includes: acquiring the external contour image of an FNA sample after paraffin embedding and staining, and a three-channel terahertz near-field detail image of the same visual region; extracting first-modal features through an improved backbone network; obtaining second-modal features aligned with the three scales of the first modality through a priori pyramid module; performing channel-level adaptive weighted fusion of the two-modal features through a gated fusion module; aggregating and optimizing the fused features through a neck module; and outputting the target detection result through a head module. Based on this, the method achieves multi-source information complementarity, reduces computational overhead through lightweight module design, enhances multi-scale feature representation capabilities, and effectively resists interference from uneven staining depth, cell overlap, and background noise, achieving high accuracy, high robustness, and efficient inference in PTC cancer cell detection under limited computing power.

[0026] like Figure 8 As shown in the embodiments of this application, the multimodal detection method for papillary thyroid carcinoma cells based on YOLOv11 includes: S801, Obtain the first image and the second image.

[0027] The first image is the outline of a thyroid fine-needle aspiration (FNA) sample after paraffin embedding and staining; the second image is a three-channel terahertz near-field internal detail image of the same field of view.

[0028] In this embodiment of the application, a sample of suspected thyroid lesion cells was obtained by FNA. After fixation, dehydration, paraffin embedding, sectioning and staining, an atomic force microscope was used to perform morphological imaging to obtain an external outline image, i.e., the first image. A terahertz near-field imaging system was used to perform near-field scanning imaging on the same field of view section to obtain an internal detail image, i.e., the second image.

[0029] It should be noted that the two images must be precisely aligned in terms of their fields of view to avoid spatial misalignment affecting the subsequent feature fusion effect.

[0030] As an example, an external outline image is obtained using a wafer-level atomic force microscope (AtomMax), and an internal detail image is obtained using a three-dimensional terahertz imaging scanner.

[0031] Based on the above steps, pixel-level aligned images of two modalities are obtained, which improves the robustness of sample differences.

[0032] S802. Based on the improved backbone network, multi-scale feature extraction is performed on the first image to obtain the first modality features.

[0033] The first modality features are the feature information of the first image at three scales: P3, P4, and P5. The improved backbone network integrates the C3k2_Faster module and the C2MSLA module.

[0034] In this embodiment, after receiving the first image, the improved backbone network reduces computational overhead through the C3k2_Faster module, while strengthening multi-scale texture extraction and context modeling through the C2MSLA module in the high semantic layer, extracting feature information of different scales layer by layer, and finally outputting the three-scale first modality features.

[0035] It should be noted that the feature extraction process must maintain the semantic integrity and spatial consistency of features at each scale.

[0036] As an example, the C3k2_Faster module concatenates two Faster_Block blocks, and the C2MSLA module concatenates two MSLAlock blocks, ultimately outputting three-scale first-mode features, including first-mode features at the P3 scale. First modal features at the P4 scale First modal features at the P5 scale .

[0037] Based on the above steps, multi-scale features with both lightweight and strong representational capabilities are obtained, providing high-quality input for subsequent cross-modal fusion.

[0038] S803. Based on the prior pyramid module, perform stem processing and progressive downsampling on the second image to obtain the second modality features.

[0039] The second modality features are feature information that are aligned with the scales of the first modality, P3, P4, and P5.

[0040] In this embodiment, the prior pyramid module takes the input tensor corresponding to the second image as input, first filters noise and compresses dimensions through STEM to obtain an initial feature map, then generates a three-scale intermediate feature map through multiple rounds of progressive downsampling, and finally outputs a second modal feature aligned with the scale of the first modality after channel adjustment and scale adaptation.

[0041] It should be noted that the downsampling process requires precise control of the scale factor to ensure spatial matching with the first modality features.

[0042] Based on the above steps, the scale of the second modality feature and the first modality feature are precisely aligned.

[0043] S804. The first modal feature and the second modal feature are subjected to channel-level adaptive weighted fusion through the gated fusion module, and the fused feature is output.

[0044] Among them, channel-level adaptive weighted fusion refers to the fusion of features by dynamically allocating weights based on the semantic relevance of the two modalities.

[0045] In this embodiment, the gated fusion module first extracts and splices global information of the two modal features at each scale corresponding to P3, P4, and P5, then generates adaptive gated weights through feature transformation, and finally performs channel-wise weighted fusion of the two modal features based on these weights to output fused features.

[0046] It should be noted that the fusion process needs to be executed independently at each scale to ensure the targeted fusion of features at different scales.

[0047] Based on the above steps, the complementary information of the two modalities is fused to improve the feature's adaptability to complex scenarios.

[0048] S805. Aggregate and optimize the fused features based on the neck module.

[0049] Among them, aggregation optimization refers to strengthening the correlation and integrity of multi-scale features through feature transfer and interaction.

[0050] In this embodiment, the neck module adopts a combination structure of FPN and PAN. The FPN structure realizes the top-down semantic information transmission of fused features, and the PAN structure realizes the bottom-up aggregation of detailed information. Through bidirectional interaction, the deep optimization of multi-scale fused features is completed.

[0051] It should be noted that the feature transfer and aggregation processes must maintain consistency in feature dimensions.

[0052] Based on the above steps, the correlation and complementarity of features at different scales are strengthened, and the detection adaptability to cell targets of different sizes is improved.

[0053] S806. The head module outputs the target bounding box and prediction confidence based on the aggregated and optimized fusion features to obtain the target detection result.

[0054] The prediction confidence score is the probability that the target region belongs to papillary thyroid carcinoma cells, and the target detection result includes the target box location and the corresponding confidence score.

[0055] In this embodiment, the head module receives the aggregated and optimized fused features, outputs confidence scores through the classification branch of the detection head, predicts the target box coordinates and size through the regression branch, and outputs the final detection result after post-processing to filter redundant predictions.

[0056] It should be noted that the detection category is set to a single category, cancer, which simplifies the classification calculation. The model has been pre-trained using a dataset of labeled first and second images covering different pathological morphologies, cell densities, staining conditions, and background noise levels in the slice field of view. This application does not impose any specific limitations on this.

[0057] As an example, non-maximum suppression (NMS) is used to handle redundant prediction boxes.

[0058] Based on the above steps, the localization and confidence information of papillary thyroid carcinoma cells can be quickly obtained, providing a reference for image interpretation.

[0059] Based on the above technical solution, through the whole process design of multimodal data acquisition, accurate feature extraction, scale alignment, adaptive fusion and aggregation optimization, multi-source information complementarity is realized. While reducing the computational overhead, the feature representation capability is enhanced, and interference such as staining fluctuation and cell overlap is effectively resisted. Finally, under the condition of limited computing power, high accuracy, high robustness and efficient inference of detection are achieved.

[0060] In one possible implementation of the embodiments of this application, combined with Figure 8 ,like Figure 9 As shown, the above S803 can be specifically implemented through the following S901, S902 and S903, which are explained in detail below: S901. Using the second modal input tensor corresponding to the second image as input, the initial feature map is obtained through STEM processing.

[0061] The STEM process is a preprocessing operation that performs preliminary feature extraction and dimensionality compression on the original input tensor.

[0062] In this embodiment, the priori pyramid module receives the second modal input tensor. Then, through a stem that includes convolution, batch normalization, and activation functions: The input is subjected to feature mapping and spatial size compression to obtain the initial feature map. , For stem processing, For batch size, Input the number of channels. Image length, This represents the image width.

[0063] It should be noted that STEM processing needs to balance feature preservation and dimensionality compression efficiency.

[0064] As an example, a 3×3 convolution kernel with a stride of 2 is used to perform the convolution operation, and the activation function is SiLU.

[0065] Based on the above steps, redundant noise in the original image was filtered out, and initial features with basic semantic information were obtained.

[0066] S902. Perform progressive downsampling on the initial feature map sequentially to obtain the three-scale intermediate feature map.

[0067] Progressive downsampling refers to the operation of gradually reducing the spatial size of the feature map by a fixed multiple. The three scales correspond to the scale levels P3, P4, and P5.

[0068] In this embodiment, the prior pyramid module performs multiple rounds of downsampling on the initial feature map. In each round, the size is halved and the features are enhanced through a lightweight convolutional structure, gradually generating intermediate feature maps of different scales.

[0069] Optionally, multi-round progressive downsampling generates three-scale intermediate feature maps that satisfy: , , ,in, The first-round downsampling operation function consists of a 3×3 convolutional layer (stride 2), BatchNorm2d, and SiLU activation, which halves the feature map size. This is the function for the second round of downsampling, with the same structure as conv2. This is the function for the third round of downsampling, with the same structure as conv2. , , These are intermediate feature maps at sizes P3, P4, and P5, respectively.

[0070] As an example, progressive downsampling is performed in three rounds, with a downsampling factor of 2 in each round, resulting in feature maps downsampled to 1 / 8, 1 / 16, and 1 / 32.

[0071] Based on the above steps, intermediate features covering different spatial resolutions were obtained, laying the foundation for subsequent scale alignment.

[0072] S903: The three-scale intermediate feature maps are scale-adapted and channel-adjusted by a 1×1 convolutional projection layer, and the second modal features are output that are aligned with the P3, P4 and P5 scales.

[0073] The 1×1 convolutional projection layer is used to adjust the number of feature channels to achieve channel adaptation with the first modality features.

[0074] In this embodiment, the prior pyramid module configures an independent 1×1 convolutional projection layer for each scale of intermediate feature map. Through channel dimension mapping and feature calibration, the scale and number of channels of the output features are consistent with the features of the first mode P3, P4, and P5.

[0075] Optionally, channel dimension mapping and feature calibration can be used to satisfy: , , ,in, In response to A 1×1 convolutional projection layer is used to adjust the number of channels to match the P3 feature of the first mode. In response to A 1×1 convolutional projection layer is used to adjust the number of channels to match the first modality P4 feature. In response to A 1×1 convolutional projection layer is used to adjust the number of channels to match the P5 feature of the first mode. , , These are the second modal features at dimensions P3, P4, and P5, respectively.

[0076] It should be noted that the channel adjustments at each scale need to be adapted to the first modal features of the corresponding level.

[0077] Based on the above steps, the second modal features and the first modal features are accurately aligned in terms of scale and channels, meeting the requirements of cross-modal fusion.

[0078] Based on the above technical solution, while ensuring the effectiveness of features, strict scale alignment between the second modality features and the first modality features is achieved, and a lightweight convolutional structure is adopted as a whole to reduce computational overhead.

[0079] In one possible implementation of the embodiments of this application, combined with Figure 8 ,like Figure 10 As shown, the above S804 can be specifically implemented through the following S1001 to S1004, which are explained in detail below: S1001. Perform global average pooling on the first modality features and the second modality features on the same scale.

[0080] Global average pooling is an operation that calculates the average spatial dimension of each channel in the feature map, compressing spatial information while preserving channel semantics.

[0081] In this embodiment, the GAP unit of the gated fusion module performs global average pooling on the first modal features and the second modal features for each of the corresponding scales P3, P4, and P5 to obtain the corresponding pooling vectors.

[0082] Optionally, global average pooling is performed on the first modality features and the second modality features respectively to obtain the corresponding pooling vectors that satisfy the following formula:

[0083] in, This represents the first modal feature at scale k. The second modal feature is represented at scale k, and GAP(·) is the global average pooling operation. For k-scale The corresponding pooling vector, For k-scale The corresponding pooling vector, where k is the scale index. .

[0084] It should be noted that pooling operations must be performed independently within each scale and should not be cross-scaled.

[0085] As an example, for feature maps Perform global average pooling to obtain the pooling vector corresponding to size P3. .

[0086] Based on the above steps, a vector of condensed channel core information is obtained, which prepares for subsequent feature fusion.

[0087] S1002. The pooled first modal features and the second modal features are concatenated in the channel dimension to generate concatenated features.

[0088] In this embodiment, the Concat unit of the gated fusion module receives pooling vectors from both modes. and They are spliced ​​and integrated according to the channel dimension to form a spliced ​​feature containing dual-modal global information. , This is a vector concatenation operation.

[0089] It should be noted that the batch size and spatial dimension of the two modal vectors must be kept consistent during splicing.

[0090] Based on the above steps, the global channel information of the dual-modality system is integrated to provide data support for the adaptive generation of gating weights.

[0091] S1003. Perform feature transformation on the concatenated features and output the gating weights through the Sigmoid activation function.

[0092] Among them, the gating weight is a vector used to dynamically allocate the contribution ratio of the two modal features, and its value range is [0,1].

[0093] In this embodiment, the MLP unit of the gated fusion module performs dimensional transformation and nonlinear enhancement on the spliced ​​features, and then maps the features to the [0,1] interval through the Sigmoid activation function, outputting gate weights consistent with the number of feature channels.

[0094] Optionally, the gating weights are calculated according to the following formula:

[0095] in, For gating weights, It is the Sigmoid activation function. and For 1×1 convolution, For MLP Activation function.

[0096] It should be noted that the MLP unit adopts a lightweight structure to avoid increasing excessive computational overhead.

[0097] Based on the above steps, gating weights for adaptive matching of bimodal features are generated, laying the foundation for accurate fusion.

[0098] S1004. The gating weights are fused with the first modal features and the second modal features to obtain the fused features.

[0099] In this embodiment, the fusion unit of the gated fusion module multiplies the gate weights with the original features of the two modalities channel by channel, and then sums them to obtain the fused features at each scale.

[0100] Optionally, the gating weights are fused with the first modality features and the second modality features to obtain fused features that satisfy the following formula:

[0101] in, For the fusion features at scale k, This involves multiplying each channel sequentially.

[0102] It should be noted that the fusion process must strictly correspond to the features and weights of each scale to avoid scale misalignment.

[0103] Based on the above steps, channel-level adaptive weighted fusion of two modal features was achieved, highlighting complementary information.

[0104] Based on the above technical solution, the contribution of the two modal features is dynamically balanced, the complementary information of the appearance contour and internal details is effectively integrated, the adaptability of the fused features to complex scenes is improved, and the computational overhead is controlled to adapt to scenarios with limited computing power.

[0105] To highlight the effectiveness of the method in this application, as shown in Table 1, comparative experiments were conducted on this dataset using various YOLO models. Table 1 details are as follows: Table 1

[0106] Based on the table above, the detection performance (mAP50%) and parameter count comparison of different YOLO series models (v5n to v12n) on single data ("outer contour image" and "internal detail image"), as well as multimodal data ("outer contour image + internal detail image"), are presented. The mAP of each YOLO model on single data fluctuates slightly with model iteration. However, the model in this application, based on multimodal data, shows significantly higher mAP50% (61.8%) and mAP50-95% (28.5%) than the performance of each YOLO model on single data. Although the parameter count (3.2M) is slightly higher than some single-modal YOLO models, the significant improvement in detection performance is achieved through multimodal fusion. This intuitively demonstrates that this application effectively balances computational cost and detection performance.

[0107] In one possible implementation, combining Figure 8 ,like Figure 11 As shown, following S806, the YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells provided in this application embodiment further includes S1101: S1101. Ablation experiments were conducted based on an improved backbone network, a priori pyramid module, gated fusion module, neck module, and head module.

[0108] Among them, the ablation experiment is an experimental method that verifies the independent function of each module by removing, replacing or retaining the core modules of the target detection system one by one and comparing the performance of different configuration models.

[0109] In this embodiment of the application, based on the basic YOLOv11 model, multiple experimental models with different module combinations are constructed. Each model includes or excludes one or more of the improved backbone network (including C3k2_Faster and C2MSLA modules), the prior pyramid module, and the gated fusion module. The same training hyperparameters are used for training and testing on the same multimodal dataset.

[0110] It should be noted that the experimental process should maintain consistency in conditions such as dataset partitioning, number of training iterations, and optimizer to avoid irrelevant variables from interfering with the results.

[0111] As an example, multiple ablation experiments were set up to compare the model performance of traditional Concat fusion versus adding a gated fusion module, the original YOLOv11 backbone network versus replacing it with an improved backbone network, the absence of the C3k2_Faster module versus adding it, and the absence of the C2MSLA module versus adding it. The training and testing conditions were kept consistent for each group of experiments. The results show that the gated fusion module improves detection accuracy and reduces computational cost, the improved backbone network significantly reduces the number of parameters, the C3k2_Faster module improves mAP and significantly reduces GFLOPs (the number of floating-point operations per second, i.e., computational complexity), and C2MSLA improves mAP but slightly increases GFLOPs.

[0112] Based on the above steps, data on mAP, number of parameters, and computational cost corresponding to each module combination were obtained, providing experimental support for verifying the effectiveness of individual modules.

[0113] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0114] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A YOLOv11-based multimodal detection system for papillary thyroid carcinoma cells, characterized in that, include: Improved backbone network, prior pyramid module, gating fusion module, neck module and head module; The improved backbone network, by introducing the C3k2_Faster module and integrating the C2MSLA module in the high semantic layer, is used to extract and output the first modal features of the image at the three scales of P3, P4 and P5. The prior pyramid module is used to perform stem processing and progressive downsampling on the image, and outputs second modal features that are aligned with the first modal features at scales P3, P4, and P5 through convolutional projection. The gated fusion module is used to perform channel-level adaptive weighted fusion of the first modal features and the second modal features at the corresponding scale, and output the fused features; The neck module is used to transfer and aggregate the fused features; The header module is used to output the target bounding box and prediction confidence based on the aggregated and optimized fusion features to obtain the target detection result.

2. The multimodal detection system according to claim 1, characterized in that, The C3k2_Faster module includes: a first branch and a second branch; The first branch is used to preserve features of some input features after decomposition of the convolution; The second branch includes a cascaded Faster_Block, used to perform lightweight feature transformation and enhancement on another part of the input features after convolution decomposition; the Faster_Block includes a partial convolutional spatial mixing operator and a convolutional MLP and adopts a residual update mechanism.

3. The multimodal detection system according to claim 1, characterized in that, The C2MSLA module includes: a third branch and a fourth branch; The third branch is used to preserve the input features; The fourth branch includes cascaded MSLAlock blocks for multi-scale texture extraction and context modeling enhancement of input features.

4. The multimodal detection system according to claim 1, characterized in that, The gated fusion module includes: a GAP unit, a Concat unit, an MLP unit, a Sigmoid activation unit, and a fusion unit; The GAP unit is used to perform global average pooling on the first modality features and the second modality features at the same scale. The Concat unit is used to concatenate the pooled first modal feature and the second modal feature in the channel dimension to generate a concatenated feature; The MLP unit includes two layers of 1×1 convolution and a ReLU activation function, which are used to perform feature transformation on the concatenated features; The Sigmoid activation unit is used to output gate weights based on the concatenated vector after feature transformation through the Sigmoid activation function. The fusion unit is used to fuse the gating weights with the first modal features and the second modal features to obtain fused features.

5. The multimodal detection system according to claim 3, characterized in that, The MSLAlock block includes: an MSLA unit and an FFN unit; The MSLA unit is used to divide the channel into four equal parts and perform 3×3, 5×5, 7×7 and 9×9 depth convolutions respectively. The convolution results at each scale are fused and enhanced through a linear attention mechanism. At the same time, learnable scale weights are introduced to adaptively adjust the contribution of features at different scales. The FFN unit is used to perform lightweight nonlinear transformation on the multi-scale fusion features output by the MSLA unit.

6. The multimodal detection system according to claim 1, characterized in that, The neck module employs an FPN structure for top-down feature transfer of fused features and a PAN structure for bottom-up feature aggregation.

7. A YOLOv11-based multimodal detection method for papillary thyroid carcinoma cells, applied to the multimodal detection system as described in any one of claims 1-6, characterized in that, The method includes: Acquire a first image and a second image; the first image is the external outline image of the FNA sample after paraffin embedding and staining; the second image is a three-channel terahertz near-field internal detail image of the same field of view. Based on the improved backbone network, multi-scale feature extraction is performed on the first image to obtain the first modality features; Based on the prior pyramid module, the second image is processed using STEM and progressive downsampling to obtain second modal features; The gated fusion module performs channel-level adaptive weighted fusion of the first modal features and the second modal features to output fused features; The fusion features are aggregated and optimized based on the neck module; The head module outputs the target bounding box and prediction confidence based on the aggregated and optimized fusion features to obtain the target detection result.

8. The method according to claim 7, characterized in that, The step of performing STEM processing and progressive downsampling on the second image based on the prior pyramid module to obtain second modal features includes: The initial feature map is obtained by using the second modality input tensor corresponding to the second image as input and processing it through stem. The initial feature map is progressively downsampled sequentially to obtain a three-scale intermediate feature map; The three-scale intermediate feature maps are scale-adapted and channel-adjusted by a 1×1 convolutional projection layer to output second modal features aligned with the P3, P4, and P5 scales.

9. The method according to claim 7, characterized in that, The step of performing channel-level adaptive weighted fusion of the first modal features and the second modal features through the gated fusion module to output fused features includes: Global average pooling is performed on the first modality features and the second modality features at the same scale; The pooled first modal features and the second modal features are concatenated in the channel dimension to generate concatenated features; The spliced ​​features are transformed and then gating weights are output using the Sigmoid activation function. The gating weights are fused with the first modal feature and the second modal feature to obtain the fused feature.

10. The method according to claim 9, characterized in that, The gating weights are fused with the first modality features and the second modality features to obtain the fused features that satisfy the following formula: in, The fusion features at the k-scale are... The gate weights at scale k are... The first modal feature at scale k, The second modal feature at scale k. For channel-by-channel multiplication, k is the scale index. .

Images