Multi-modal ultrasonic image recognition method and device

Abstract

The invention discloses a multi-modal ultrasonic image recognition method and device, which effectively combines an ultrasonic image and a corresponding task text, and improves the recognition precision of the ultrasonic image by utilizing the interaction of visual information and language information. By preprocessing the ultrasonic image and the task text and generating a visual Token sequence and a language Token sequence, data of different modalities can be effectively fused to generate a multi-modal sequence. The sequences are subjected to context interactive fusion of a multi-head attention model to obtain a multi-modal embedded representation containing visual and language information, so that the model can accurately analyze a task instruction while understanding the image content, and the recognition accuracy is improved. And furthermore, through selection and weighted fusion of dynamic experts and in combination with background knowledge provided by static experts, the processing capability of the model is improved. And finally, the output result is adjusted and optimized by using the fusion control parameters, so that the efficiency and the accuracy of ultrasonic image recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of ultrasound recognition technology, and in particular to a multimodal ultrasound image recognition method and apparatus. Background Technology

[0002] Currently, ultrasound images exhibit significant differences in their representation across different organs, and existing unified model architectures still face performance bottlenecks in multi-task and multi-anatomical region inference. The MoE (Mixture of Experts) architecture offers a solution to improve model parameter efficiency and generalization ability. MoE decomposes the task into multiple specialized sub-models (i.e., "experts") through multimodal ultrasound task routing, and relies on a routing mechanism to dynamically select a suitable combination of experts based on the input features, thus significantly improving model performance while ensuring computational efficiency.

[0003] Existing ultrasound image recognition methods often lack static expert mechanisms in their MoE architectures, making it difficult to retain and transfer general medical knowledge in ultrasound image recognition. Since medical ultrasound images and language involve a large amount of specialized domain knowledge, relying solely on dynamic experts often fails to effectively capture the deep-seated knowledge of medical ultrasound images. Furthermore, existing methods typically fail to effectively implement semantic binding mechanisms between dynamic experts and ultrasound tasks, resulting in a lack of correlation between the selection of different expert modules and the task content. This negatively impacts the model's performance on specific tasks, leading to low accuracy in multimodal ultrasound image recognition. Summary of the Invention

[0004] This invention provides a multimodal ultrasound image recognition method and apparatus to improve the efficiency and accuracy of ultrasound image recognition.

[0005] To address the aforementioned technical problems, this invention provides a multimodal ultrasound image recognition method, comprising: Acquire ultrasound images and corresponding task texts, and preprocess the ultrasound images and corresponding task texts respectively to obtain visual token sequences and language token sequences; The language token sequence and the visual token sequence are concatenated to obtain a multimodal sequence; and the multimodal sequence is fused with contextual interaction based on a preset multi-head attention model to obtain a multimodal embedding representation. The semantic matching degree between each token in the multimodal sequence and each preset dynamic expert is calculated based on the preset neural router and the multimodal embedding representation; and several target dynamic experts and their corresponding participation weights are determined based on the semantic matching degree. The output of the target dynamic expert is weighted and fused based on the participation weight and the preset weighting method to obtain the first output result; Each token in the multimodal sequence is input in parallel into a preset static expert to obtain the second output result of the static expert; Based on preset fusion control parameters, the first output result and the second output result are fused to obtain a multimodal token representation, and an ultrasound image recognition result is generated based on the multimodal token representation.

[0006] This invention improves the recognition accuracy of ultrasound images by effectively combining ultrasound images and corresponding task text, leveraging the interaction of visual and linguistic information. By preprocessing the ultrasound images and task text separately to generate visual and linguistic token sequences, data from different modalities can be effectively fused to generate multimodal sequences. These sequences are then fused using a multi-head attention model with contextual interaction to obtain multimodal embedding representations containing both visual and linguistic information. This allows the model to understand image content while accurately parsing task instructions, improving the accuracy of the recognition results. Furthermore, by selecting and weighting fusion with dynamic experts, combined with background knowledge provided by static experts, the model's processing power is further enhanced. Finally, by adjusting and optimizing the output results using fusion control parameters, the generated ultrasound image recognition results are ensured to reflect image details, improving the reliability and accuracy of the recognition.

[0007] Furthermore, the acquisition of ultrasound images and corresponding task text, and the preprocessing of the ultrasound images and corresponding task text to obtain visual token sequences and language token sequences, include: Acquire ultrasound images and corresponding task text; The ultrasound image is subjected to target detection based on a preset target detection model to obtain a target image; and the target image is then normalized and filtered to obtain a standard image. Based on a preset visual encoder, local features of the standard image are extracted, and each local region is converted into a visual token to form a visual token sequence. The task text is segmented and embedded to generate a language token sequence.

[0008] This invention identifies key regions in images through a target detection model and performs normalization and filtering processes, ensuring the standardization and clarity of the input images, thus providing high-quality image data for subsequent visual feature extraction. This step effectively removes irrelevant information, highlights key diagnostic regions in the image, and improves the accuracy of image feature extraction. Based on a visual encoder, local feature extraction is performed on the standard image, converting each local region into a visual token, ensuring that each visual token accurately reflects the detailed features of the image. Furthermore, word segmentation and embedding of the task text lay the foundation for generating language token sequences, ensuring the alignment of task instructions with image information, thereby better supporting multimodal fusion.

[0009] Furthermore, the step of performing word segmentation and embedding processing on the task text to generate a language token sequence includes: The task text is segmented to obtain the segmented words in the task text; The word segments are embedded to convert each word into a corresponding language token, and a language token sequence is formed based on each language token.

[0010] This invention employs word segmentation to divide text into semantic units, ensuring that each token accurately represents the information within the text. Subsequently, embedding processing transforms these segments into corresponding language tokens and generates a sequence of language tokens, providing the model with processable linguistic feature representations. This process guarantees efficient expression of textual information, provides clear and consistent input for multimodal models, improves the information alignment between images and task text during fusion, and contributes to enhancing the accuracy of ultrasound image recognition.

[0011] Furthermore, the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert is calculated based on the preset neural router and the multimodal embedding representation; and several target dynamic experts and their corresponding participation weights are determined based on the semantic matching degree, including: Each token of the multimodal embedding representation is input into the neural router to extract the semantic representation of each token; Based on a preset similarity calculation algorithm, the semantic representation of each token is similar to the preset semantic representation of each dynamic expert to obtain the semantic matching degree between each token and each dynamic expert. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined.

[0012] This invention effectively captures the semantic information of each token by inputting it into a neural router and extracting its semantic representation. This step, through similarity calculation, accurately measures the matching degree between each token and a dynamic expert, making expert selection more precise. Using a pre-defined similarity calculation algorithm, the model can efficiently evaluate the relationship between tokens and experts, thus providing a scientific basis for subsequent dynamic expert selection. Through this refined semantic matching, the model can better understand the needs of different tasks and images, and select the most suitable expert based on different task objectives and image features, thereby improving the accuracy and efficiency of the recognition process.

[0013] Furthermore, determining several target dynamic experts and their corresponding participation weights based on the semantic matching degree includes: The importance score of each dynamic expert is determined based on the semantic matching degree, and the dynamic experts are sorted in descending order based on the importance score to generate a dynamic expert list. Based on preset values, select several dynamic experts with the highest importance scores from the dynamic expert list as target dynamic experts, and calculate the corresponding participation weights based on the importance scores of the target dynamic experts.

[0014] This invention determines the importance score of dynamic experts based on semantic matching degree and ranks these experts in descending order, enabling the model to prioritize experts most relevant to the task. A participation weight is calculated for each target expert based on the matching degree score, ensuring that the most relevant experts receive more weight and undergo deeper processing. This process effectively optimizes expert selection and weight allocation, ensuring that each token receives the most appropriate expert processing, thereby improving the accuracy and robustness of the entire multimodal ultrasound image recognition method. Simultaneously, the expert selection strategy based on preset values ​​guarantees the system's efficiency and controllability, avoiding waste of computational resources.

[0015] In a second aspect, the present invention provides a multimodal ultrasound image recognition device, comprising: a data preprocessing module, a fusion module, a dynamic expert matching module, a first output module, a second output module, and a recognition module; The data preprocessing module is used to acquire ultrasound images and corresponding task texts, and preprocess the ultrasound images and corresponding task texts respectively to obtain visual token sequences and language token sequences. The fusion module is used to concatenate the language token sequence and the visual token sequence to obtain a multimodal sequence; and to perform contextual interaction fusion on the multimodal sequence based on a preset multi-head attention model to obtain a multimodal embedding representation. The dynamic expert matching module is used to calculate the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert based on the preset neural router and the multimodal embedding representation; and to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree. The first output module is used to perform weighted fusion of the output of the target dynamic expert based on the participation weight and a preset weighting method to obtain a first output result; The second output module is used to input each token in the multimodal sequence into a preset static expert in parallel to obtain the second output result of the static expert; The recognition module is used to fuse the first output result and the second output result based on preset fusion control parameters to obtain a multimodal token representation, and generate an ultrasound image recognition result based on the multimodal token representation.

[0016] Furthermore, the data preprocessing module is used to acquire ultrasound images and corresponding task text, and to preprocess the ultrasound images and corresponding task text respectively to obtain visual token sequences and language token sequences, including: Acquire ultrasound images and corresponding task text; The ultrasound image is subjected to target detection based on a preset target detection model to obtain a target image; and the target image is then normalized and filtered to obtain a standard image. Based on a preset visual encoder, local features of the standard image are extracted, and each local region is converted into a visual token to form a visual token sequence. The task text is segmented and embedded to generate a language token sequence.

[0017] Furthermore, the data preprocessing module is used to perform word segmentation and embedding processing on the task text to generate a language token sequence, including: The task text is segmented to obtain the segmented words in the task text; The word segments are embedded to convert each word into a corresponding language token, and a language token sequence is formed based on each language token.

[0018] Furthermore, the dynamic expert matching module is used to calculate the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert based on the preset neural router and the multimodal embedding representation; and to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree, including: Each token of the multimodal embedding representation is input into the neural router to extract the semantic representation of each token; Based on a preset similarity calculation algorithm, the semantic representation of each token is similar to the preset semantic representation of each dynamic expert to obtain the semantic matching degree between each token and each dynamic expert. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined.

[0019] Furthermore, the dynamic expert matching module is used to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree, including: The importance score of each dynamic expert is determined based on the semantic matching degree, and the dynamic experts are sorted in descending order based on the importance score to generate a dynamic expert list. Based on preset values, select several dynamic experts with the highest importance scores from the dynamic expert list as target dynamic experts, and calculate the corresponding participation weights based on the importance scores of the target dynamic experts. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a multimodal ultrasound image recognition method provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart of a multimodal ultrasound image recognition device provided in an embodiment of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] The terms "first" and "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Example 1 See Figure 1 , Figure 1 This is a flowchart illustrating a multimodal ultrasound image recognition method according to an embodiment of the present invention. The embodiment of the present invention provides a multimodal ultrasound image recognition method, including steps 101 to 106, as detailed below: Step 101: Acquire ultrasound images and corresponding task texts, and preprocess the ultrasound images and corresponding task texts respectively to obtain visual token sequences and language token sequences; In this embodiment, the acquisition of ultrasound images and corresponding task text, and the preprocessing of the ultrasound images and corresponding task text to obtain visual token sequences and language token sequences, include: Acquire ultrasound images and corresponding task text; The ultrasound image is subjected to target detection based on a preset target detection model to obtain a target image; and the target image is then normalized and filtered to obtain a standard image. Based on a preset visual encoder, local features of the standard image are extracted, and each local region is converted into a visual token to form a visual token sequence. The task text is segmented and embedded to generate a language token sequence.

[0025] In this embodiment, firstly, ultrasound images of the target patient and corresponding task text are acquired. The task text typically includes structured instructions related to the ultrasound images, such as "Please generate a cardiac ultrasound report" or "Please generate a breast ultrasound report." Subsequently, preprocessing is performed on both the ultrasound images and the task text.

[0026] In this embodiment, ultrasound images, as a type of medical imaging, typically contain various anatomical structures and lesion targets, such as structures of the breast, thyroid, liver, gallbladder, heart, and urinary system, or various lesions such as nodules, masses, and cysts. The required task text differs for each different target type. Different task texts are generated based on preset task requirements for different target types. For example, when processing breast ultrasound images, the task text might include "Please confirm the presence of breast nodules" or "Is there a mass?". When processing thyroid ultrasound images, the task text might become "Please confirm the location and size of thyroid nodules" or "Are there any signs of thyroid dysfunction?" These task texts are closely related to the specific target type, guiding the system to focus on specific areas and pathological features in the image.

[0027] In practice, ultrasound images are first processed by a target detection model to identify and locate key target regions (such as nodules and masses) within the image. These regions are then further analyzed based on the requirements of the task text. The task text is generated according to clinical needs and input into the system along with the corresponding ultrasound images. Guided by the task text, the target type and the direction of image processing are clearly defined, thereby enabling accurate identification and analysis of different target regions in the ultrasound images.

[0028] In the preprocessing of ultrasound images, a pre-defined target detection model, such as YOLOv3, is first used to detect targets in key regions of the ultrasound image, obtaining target images containing diagnostic significance. These key regions may contain important anatomical structures, such as masses, nodules, or other lesions. Next, the detected target images are normalized to ensure that the image size and grayscale values ​​meet the standards for subsequent processing. Simultaneously, noise filtering is performed to remove interfering information from the image, ultimately obtaining a standard image.

[0029] In this embodiment, ultrasound images may contain sensitive identity information such as the patient's name, examination number, and hospital name. This information is usually located in the edge areas of the image, while information directly related to the ultrasound recognition task is located in the center or specific areas of the image. To protect user privacy and prevent the leakage of sensitive information, the system needs to automatically identify and remove these areas containing identity information, retaining only the key areas in the target image.

[0030] In this embodiment, a preset visual encoder is used to extract local features based on a standard image. The visual encoder segments the standard image into multiple small regions and extracts feature vectors for each region. Each feature vector represents information about a local region in the image. These feature vectors are converted into visual tokens, with each token corresponding to a local region in the image, forming a complete sequence of visual tokens.

[0031] In this embodiment, a token typically refers to the smallest unit of text or image: In one optional embodiment, each image has its features extracted using a pre-trained CLIP (Contrastive Language-Image Pretraining) visual encoder to obtain a patch-level visual token sequence. Typically, several patch tokens are extracted from each image, representing local region information. Patch-level refers to a hierarchy in image processing where image patches are the basic processing units. Each patch represents a local region in the image.

[0032] In an optional embodiment, to reduce the input size and computational burden, the PatchMerger module is used to perform cross-image fusion of tokens from multiple images, such as merging and compressing tokens every 4 patches.

[0033] In an optional embodiment, all visual tokens are projected into the embedding space of the language model via a lightweight multilayer perceptron projection module to support visual and language fusion learning. This module supports LoRA insertion for easy fine-tuning later.

[0034] In this embodiment, key regions in the image are identified by an object detection model and then normalized and filtered to ensure the standardization and clarity of the input image, thus providing high-quality image data for subsequent visual feature extraction. This step effectively removes irrelevant information, highlights key diagnostic regions in the image, and improves the accuracy of image feature extraction. Local feature extraction is performed on the standard image based on a visual encoder, converting each local region into a visual token, ensuring that each visual token accurately reflects the detailed features of the image. Furthermore, word segmentation and embedding of the task text lay the foundation for generating a language token sequence, ensuring the alignment of task instructions with image information, thereby better supporting multimodal fusion.

[0035] In this embodiment, the step of performing word segmentation and embedding processing on the task text to generate a language token sequence includes: The task text is segmented to obtain the segmented words in the task text; The word segments are embedded to convert each word into a corresponding language token, and a language token sequence is formed based on each language token.

[0036] In this embodiment, the input task text is segmented into words. The purpose of word segmentation is to divide the task text into smaller units, such as words or sub-words, which have independent semantic meanings. For example, the input text "Does breast nodules exist?" may be broken down into word units such as "whether", "exist", "breast", and "nodules" after word segmentation. The segmented units provide the foundation for subsequent semantic embedding, enabling each component of the text to be understood and processed independently.

[0037] In this embodiment, an embedding method is used to vectorize each unit after word segmentation. Through embedding, each word segment is transformed into a fixed-dimensional language token, typically obtained using a pre-trained word vector model. The vector of each language token contains the position of the word in the semantic space, effectively representing the semantic features of the word segment. For example, the word "breast" will be converted into a vector representing the semantic information of the word.

[0038] In this embodiment, each word in the task text is converted into a corresponding language token, thus forming a complete language token sequence. This language token sequence provides semantic support for subsequent multimodal sequence splicing and context fusion, enabling the system to effectively combine the task text with the visual features of ultrasound images, thereby achieving accurate task execution and image analysis.

[0039] In this embodiment, word segmentation divides the text into semantic units, ensuring that each token accurately represents the information within the text. Subsequently, embedding processing converts these segments into corresponding language tokens and generates a sequence of language tokens, providing the model with processable linguistic feature representations. This process guarantees efficient expression of textual information, provides clear and consistent input to the multimodal model, improves the information alignment between images and task text during fusion, and contributes to enhancing the accuracy of ultrasound image recognition.

[0040] Step 102: Concatenate the language token sequence and the visual token sequence to obtain a multimodal sequence; and perform contextual interaction fusion on the multimodal sequence based on a preset multi-head attention model to obtain a multimodal embedding representation; In this embodiment, for the ultrasound image recognition task, a multimodal embedding representation is obtained by concatenating linguistic and visual token sequences and performing contextual interaction fusion based on a preset multi-head attention model. By fusing visual and linguistic information, the system can comprehensively understand ultrasound images and task instructions, thereby improving the accuracy of ultrasound image recognition.

[0041] In this embodiment, firstly, the visual token sequence extracted from the ultrasound image is concatenated with the language token sequence generated from the task text. The visual token sequence represents the features of various local regions in the image, while the language token sequence contains task instructions related to the image. The concatenation process combines these two different modalities of data into a unified multimodal sequence, enabling the system to process image and text information simultaneously, thereby achieving multimodal learning.

[0042] In this embodiment, a pre-defined multi-head attention model is used to perform contextual interaction fusion on the concatenated multimodal sequence. The multi-head attention model establishes associations between each token in the multimodal sequence through a self-attention mechanism, considering the mutual influence and dependencies between different tokens. That is, visual tokens and linguistic tokens interact through an attention mechanism, enabling each token to not only focus on its own information but also on other related modal information, such as the relationship between image regions and task instructions. This contextual interaction fusion allows visual and linguistic information to complement each other, thereby generating a richer multimodal representation.

[0043] In this embodiment, the system can effectively capture the relationship between visual and linguistic information, thereby improving the ability to understand ultrasound images. During task execution, multimodal embedding representation serves as the model input, supporting subsequent task reasoning and image analysis, ensuring the accuracy and robustness of the recognition results.

[0044] In this embodiment, by concatenating language token sequences and visual token sequences and utilizing a multi-head attention model for contextual interaction fusion, visual and language information can be effectively fused deeply, improving the multimodal perception capability of ultrasound image recognition. This process enables the system to simultaneously understand image content and task instructions, thereby better performing medical image analysis tasks.

[0045] Step 103: Calculate the semantic matching degree between each Token in the multimodal sequence and each preset dynamic expert based on the preset neural router and the multimodal embedding representation; and determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree; In this embodiment, the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert is calculated based on the preset neural router and the multimodal embedding representation; and several target dynamic experts and their corresponding participation weights are determined based on the semantic matching degree, including: Each token of the multimodal embedding representation is input into the neural router to extract the semantic representation of each token; Based on a preset similarity calculation algorithm, the semantic representation of each token is similar to the preset semantic representation of each dynamic expert to obtain the semantic matching degree between each token and each dynamic expert. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined.

[0046] In this embodiment, a lightweight neural router is used to calculate the semantic matching degree between each token and all dynamic experts (including organ experts and shared experts) to form routing logits. The dynamic experts are obtained through supervised training on multi-organ, multi-task ultrasound data, and each expert module corresponds to a specific anatomical location or diagnostic task.

[0047] In this embodiment, the neural router is a lightweight multilayer perceptron that receives each input token and generates corresponding routing logits (importance scores) by extracting its semantic representation. The core function of this router is to calculate the semantic matching degree between each token and each dynamic expert based on the feature representation of the input token. These scores reflect the degree of matching between each token and the expert module and serve as weight coefficients for token allocation to dynamic experts. Based on this, a top-k strategy is applied to select the top k dynamic experts with the highest scores from the routing logits, ensuring that each token enters at most k expert paths, thereby reducing redundant computation and improving system efficiency. Through this strategy, the system can effectively allocate tokens to the most relevant expert paths, optimizing the utilization of computational resources.

[0048] In this embodiment, the matching score of each token on its target dynamic expert is processed using Softmax normalization to obtain the participation weight of each target dynamic expert. These weights represent the participation ratio of each expert in token processing, that is, the contribution of each expert to that token. Finally, based on these calculation results, several target dynamic experts can be dynamically selected and assigned corresponding participation weights. This context-based dynamic expert selection mechanism can flexibly allocate expert resources according to different task requirements, thereby improving the diagnostic accuracy and computational efficiency of the model. In this embodiment, by inputting each token into the neural router and extracting its semantic representation, the system can effectively capture the semantic information of each token. This step accurately measures the matching degree between each token and the dynamic expert through similarity calculation, making expert selection more accurate. Using a preset similarity calculation algorithm, the model can efficiently evaluate the relationship between tokens and experts, thereby providing a scientific basis for subsequent dynamic expert selection. Through this refined semantic matching, the model can better understand the needs of different tasks and images, and select the most suitable expert according to different task objectives and image features, thereby improving the accuracy and efficiency of the recognition process.

[0049] In this embodiment, determining several target dynamic experts and their corresponding participation weights based on the semantic matching degree includes: The importance score of each dynamic expert is determined based on the semantic matching degree, and the dynamic experts are sorted in descending order based on the importance score to generate a dynamic expert list. Based on preset values, select several dynamic experts with the highest importance scores from the dynamic expert list as target dynamic experts, and calculate the corresponding participation weights based on the importance scores of the target dynamic experts.

[0050] In one optional embodiment, based on the semantic matching degree, an importance score is first calculated for each dynamic expert. This score can be obtained by aggregating the matching degree of the expert from all tokens, thereby reflecting the overall relevance of the expert relative to the current multimodal input. Then, all dynamic experts are sorted in descending order according to the importance score to generate a dynamic expert list. According to a preset top-k strategy, several experts with the highest scores are selected from the list as target dynamic experts. For the selected target dynamic experts, the corresponding participation weight is calculated based on their importance score. The calculation method can adopt a normalized ratio or a Softmax form with a temperature parameter to ensure that the weights are non-negative and sum to 1. Experts with minimal contributions can be further removed by pruning or minimum weight threshold. Finally, these participation weights are assigned to the corresponding expert paths as coefficients for subsequent expert module fusion or weighted output to realize dynamic expert selection and weighted fusion based on the current context. In the implementation, deterministic parallel processing rules or randomization strategies can be introduced to handle cases with the same or similar scores, ensuring system stability and repeatability.

[0051] In this embodiment, by determining the importance score of dynamic experts based on semantic matching degree and ranking these experts in descending order, the model can prioritize selecting the experts most relevant to the task. Participation weights are calculated for each target expert based on the matching degree score, ensuring that the most relevant experts receive more weight and undergo deeper processing. This process effectively optimizes expert selection and weight allocation, enabling each token to receive the most suitable expert processing, thereby improving the accuracy and robustness of the entire multimodal ultrasound image recognition method. Simultaneously, the expert selection strategy based on preset values ​​ensures the system's efficiency and controllability, avoiding waste of computational resources.

[0052] Step 104: Based on the participation weights and the preset weighting method, perform weighted fusion on the output of the target dynamic expert to obtain the first output result; In this embodiment, a weighted fusion strategy based on pre-calculated participation weights is applied to the outputs of several target dynamic experts determined by the router to obtain the first output result. Specifically, let the selected set of target dynamic experts be {E1, ..., E...} k The corresponding participation weights are {w1, ..., w}. k For each input token, first ensure that the output vectors of each dynamic expert are aligned in dimension (mapped and standardized using a linear projection layer or layer normalization if necessary), then perform element-wise weighted summation according to the weights, formally represented as: (1) in, This represents the output representation of the token by the i-th target dynamic expert. This is the first output result.

[0053] In this embodiment, to ensure numerical stability and robustness, temperature scaling or threshold pruning can be applied to the weights before weighting, and Dropout and LayerNorm can be applied after fusion to suppress overfitting and unify the representation scale. Furthermore, a lightweight residual or scaling factor can be optionally introduced into each expert output to correct expert bias, thereby improving the semantic consistency of the fused representation and its adaptability to downstream tasks. This "first output result" will be used for further fusion with the static expert output according to the preset fusion control parameter γ, or directly entered into the multi-task head for task-specific inference.

[0054] In this embodiment, the dynamic expert module is obtained through supervised end-to-end training.

[0055] In this embodiment, training samples are constructed based on an organ-balanced multi-task ultrasound dataset. Each sample contains a multimodal input sequence (visual token and language token) and corresponding labels. During the forward propagation stage, the input is fed into the neural router after context fusion. The router calculates routing logits for each token and generates expert participation weights. Then, based on these participation weights, the outputs of several selected dynamic experts are weighted and fused to obtain dynamic expert outputs. At the same time, static expert outputs are obtained in parallel and merged with dynamic expert outputs according to the fusion control parameter γ for downstream task heads to calculate prediction results. Subsequently, an overall loss function is constructed based on the task labels.

[0056] In this embodiment, the loss consists of a main task loss term and several regularization terms, including expert load balancing loss, routing regularization, and parameter regularization. A weighted multi-task loss strategy is used for different tasks when necessary. During the backpropagation phase, gradients are propagated back and updated along the parameters of the feedforward network within the dynamic expert module and the router (matching logits generation network). The static expert module remains frozen and does not receive parameter updates. Furthermore, LoRA (low-rank adaptation) can be used in the visual projection layer or large model layer to update only low-rank parameters, reducing fine-tuning costs. Training employs a conventional optimizer combined with learning rate scheduling and gradient pruning techniques to ensure stability. The main task metrics and expert utilization are monitored on the validation set for early stopping and hyperparameter adjustment.

[0057] In this embodiment, through the above process, dynamic experts and routers collaboratively learn an expert set that can both achieve task specialization and complement static general knowledge, thereby achieving good generalization and controllability in multi-organ, multi-task clinical ultrasound scenarios.

[0058] Step 105: Input each Token in the multimodal sequence into a preset static expert in parallel to obtain the second output result of the static expert; In this embodiment, each token in the multimodal input sequence is fed in parallel into a pre-trained and parameter-frozen static expert module to compute a general representation, thereby obtaining a second output result.

[0059] In this embodiment, the static expert is typically composed of a multilayer perceptron (MLP) or several layers of feedforward network, and its input maintains the same embedding dimension as the dynamic expert. To ensure numerical consistency, LayerNorm or linear mapping can be applied to project the tokens into the input space of the static expert before inputting them into the static expert. All tokens are input in parallel batch processing, and the static expert independently infers for each token and outputs the corresponding vector representation, forming an output sequence that corresponds one-to-one with the input sequence.

[0060] In this embodiment, static expert parameters remain frozen during training, and their output primarily carries general medical prior knowledge obtained through large-scale medical image and text pre-training. This knowledge provides stable general information support in subsequent steps of weighted fusion with dynamic experts. To adapt to downstream fusion and task headers, the static expert output can be supplemented with small linear projections or LayerNorms when necessary to unify the scale and improve fusion robustness.

[0061] Step 106: Based on preset fusion control parameters, fuse the first output result and the second output result to obtain a multimodal token representation, and generate an ultrasound image recognition result based on the multimodal token representation.

[0062] In this embodiment, the dynamic expert representation is obtained by weighted fusion of several target dynamic experts according to their participation weights. The static expert representation obtained by parallel computing Token-by-token weighted fusion is performed according to the preset fusion control parameter γ to generate a multimodal final representation for each token. Specifically, for the j-th token in the sequence, we have: (2) Where γ∈[0,1] can be a global scalar, a vector defined by layer or head, or further extended to learnable token / channel-level weights to achieve a dynamic balance between general knowledge and task-specific capabilities during training. To ensure numerical stability and representation consistency, linear mapping (projection), LayerNorm, and optional Dropout can be applied to the outputs of each branch before and after fusion. The fusion result can then be processed by a lightweight feedforward network (or residual connection + LayerNorm) for nonlinear mapping and scale correction to obtain the final multimodal token representation sequence {ymm,1,…,ymm,n}. Subsequently, this multimodal token representation is input to the downstream recognition head, and the ultrasound image recognition result is calculated through the corresponding task-specific network structure and loss function. Based on this, subsequent confidence scoring is performed, and a recognition report is generated.

[0063] In this embodiment, the recognition head includes a classification head, a bounding box regression head, a pixel-level segmentation head, or a report generation decoder based on autoregression or attention.

[0064] Please refer to Figure 2 , Figure 2 The present invention provides a schematic flowchart of a multimodal ultrasound image recognition device, comprising: a data preprocessing module 201, a fusion module 202, a dynamic expert matching module 203, a first output module 204, a second output module 205, and a recognition module 206; The data preprocessing module 201 is used to acquire ultrasound images and corresponding task texts, and preprocess the ultrasound images and corresponding task texts respectively to obtain visual token sequences and language token sequences. The fusion module 202 is used to concatenate the language token sequence and the visual token sequence to obtain a multimodal sequence; and to perform contextual interaction fusion on the multimodal sequence based on a preset multi-head attention model to obtain a multimodal embedding representation; The dynamic expert matching module 203 is used to calculate the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert based on the preset neural router and the multimodal embedding representation; and to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree. The first output module 204 is used to perform weighted fusion of the output of the target dynamic expert based on the participation weight and the preset weighting method to obtain a first output result; The second output module 205 is used to input each Token in the multimodal sequence into a preset static expert in parallel to obtain the second output result of the static expert; The recognition module 206 is used to fuse the first output result and the second output result based on preset fusion control parameters to obtain a multimodal token representation, and generate an ultrasound image recognition result based on the multimodal token representation.

[0065] In this embodiment, the data preprocessing module is used to acquire ultrasound images and corresponding task text, and to preprocess the ultrasound images and corresponding task text to obtain visual token sequences and language token sequences, including: Acquire ultrasound images and corresponding task text; The ultrasound image is subjected to target detection based on a preset target detection model to obtain a target image; and the target image is then normalized and filtered to obtain a standard image. Based on a preset visual encoder, local features of the standard image are extracted, and each local region is converted into a visual token to form a visual token sequence. The task text is segmented and embedded to generate a language token sequence.

[0066] In this embodiment, the data preprocessing module is used to perform word segmentation and embedding processing on the task text to generate a language token sequence, including: The task text is segmented to obtain the segmented words in the task text; The word segments are embedded to convert each word into a corresponding language token, and a language token sequence is formed based on each language token.

[0067] In this embodiment, the dynamic expert matching module is used to calculate the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert based on a preset neural router and the multimodal embedding representation; and to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree, including: Each token of the multimodal embedding representation is input into the neural router to extract the semantic representation of each token; Based on a preset similarity calculation algorithm, the semantic representation of each token is similar to the preset semantic representation of each dynamic expert to obtain the semantic matching degree between each token and each dynamic expert. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined.

[0068] In this embodiment, the dynamic expert matching module is used to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree, including: The importance score of each dynamic expert is determined based on the semantic matching degree, and the dynamic experts are sorted in descending order based on the importance score to generate a dynamic expert list. Based on preset values, select several dynamic experts with the highest importance scores from the dynamic expert list as target dynamic experts, and calculate the corresponding participation weights based on the importance scores of the target dynamic experts.

[0069] In this embodiment of the invention, a terminal device is also provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described multimodal ultrasound image recognition method.

[0070] In this embodiment of the invention, a computer-readable storage medium is also provided, which includes a stored computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the above-described multimodal ultrasound image recognition method when it is running.

[0071] For example, a computer program can be divided into one or more modules, one or more of which are stored in memory and executed by a processor to perform the present invention. The one or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a terminal device.

[0072] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor, memory, and display. Those skilled in the art will understand that the above components are merely examples of terminal devices and do not constitute a limitation on the terminal device. It may include more or fewer components, or combinations of certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0073] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device through various interfaces and lines.

[0074] Memory can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound playback, text conversion, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0075] The module for multimodal ultrasound image recognition, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. Those skilled in the art can understand and implement this without any inventive effort.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A multimodal ultrasound image recognition method, characterized in that, include: Acquire ultrasound images and corresponding task texts, and preprocess the ultrasound images and corresponding task texts respectively to obtain visual token sequences and language token sequences; The language token sequence and the visual token sequence are concatenated to obtain a multimodal sequence; The multimodal sequence is then fused with contextual interaction based on a preset multi-head attention model to obtain a multimodal embedding representation; The semantic matching degree between each token in the multimodal sequence and each preset dynamic expert is calculated based on the preset neural router and the multimodal embedding representation. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined; The output of the target dynamic expert is weighted and fused based on the participation weight and the preset weighting method to obtain the first output result; Each token in the multimodal sequence is input in parallel into a preset static expert to obtain the second output result of the static expert; Based on preset fusion control parameters, the first output result and the second output result are fused to obtain a multimodal token representation, and an ultrasound image recognition result is generated based on the multimodal token representation.

2. The multimodal ultrasound image recognition method as described in claim 1, characterized in that, The process of acquiring ultrasound images and corresponding task text, and preprocessing the ultrasound images and corresponding task text to obtain visual token sequences and language token sequences, includes: Acquire ultrasound images and corresponding task text; The ultrasound image is subjected to target detection based on a preset target detection model to obtain a target image; and the target image is then normalized and filtered to obtain a standard image. Based on a preset visual encoder, local features of the standard image are extracted, and each local region is converted into a visual token to form a visual token sequence. The task text is segmented and embedded to generate a language token sequence.

3. The multimodal ultrasound image recognition method as described in claim 2, characterized in that, The step of segmenting and embedding the task text to generate a language token sequence includes: The task text is segmented to obtain the segmented words in the task text; The word segments are embedded to convert each word into a corresponding language token, and a language token sequence is formed based on each language token.

4. The multimodal ultrasound image recognition method as described in claim 3, characterized in that, The semantic matching degree between each token in the multimodal sequence and each preset dynamic expert is calculated based on the preset neural router and the multimodal embedding representation. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined, including: Each token of the multimodal embedding representation is input into the neural router to extract the semantic representation of each token; Based on a preset similarity calculation algorithm, the semantic representation of each token is similar to the preset semantic representation of each dynamic expert to obtain the semantic matching degree between each token and each dynamic expert. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined.

5. The multimodal ultrasound image recognition method as described in claim 4, characterized in that, The determination of several target dynamic experts and their corresponding participation weights based on the semantic matching degree includes: The importance score of each dynamic expert is determined based on the semantic matching degree, and the dynamic experts are sorted in descending order based on the importance score to generate a dynamic expert list. Based on preset values, select several dynamic experts with the highest importance scores from the dynamic expert list as target dynamic experts, and calculate the corresponding participation weights based on the importance scores of the target dynamic experts.

6. A multimodal ultrasound image recognition device, characterized in that, include: The system includes a data preprocessing module, a fusion module, a dynamic expert matching module, a first output module, a second output module, and a recognition module. The data preprocessing module is used to acquire ultrasound images and corresponding task texts, and preprocess the ultrasound images and corresponding task texts respectively to obtain visual token sequences and language token sequences. The fusion module is used to concatenate the language token sequence and the visual token sequence to obtain a multimodal sequence; The multimodal sequence is then fused with contextual interaction based on a preset multi-head attention model to obtain a multimodal embedding representation; The dynamic expert matching module is used to calculate the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert based on the preset neural router and the multimodal embedding representation; Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined; The first output module is used to perform weighted fusion of the output of the target dynamic expert based on the participation weight and a preset weighting method to obtain a first output result; The second output module is used to input each token in the multimodal sequence into a preset static expert in parallel to obtain the second output result of the static expert; The recognition module is used to fuse the first output result and the second output result based on preset fusion control parameters to obtain a multimodal token representation, and generate an ultrasound image recognition result based on the multimodal token representation.

7. The multimodal ultrasound image recognition device as described in claim 6, characterized in that, The data preprocessing module is used to acquire ultrasound images and corresponding task text, and to preprocess the ultrasound images and corresponding task text to obtain visual token sequences and language token sequences, including: Acquire ultrasound images and corresponding task text; The ultrasound image is subjected to target detection based on a preset target detection model to obtain a target image; and the target image is then normalized and filtered to obtain a standard image. Based on a preset visual encoder, local features of the standard image are extracted, and each local region is converted into a visual token to form a visual token sequence. The task text is segmented and embedded to generate a language token sequence.

8. The multimodal ultrasound image recognition device as described in claim 7, characterized in that, The data preprocessing module is used to perform word segmentation and embedding processing on the task text to generate a language token sequence, including: The task text is segmented to obtain the segmented words in the task text; The word segments are embedded to convert each word into a corresponding language token, and a language token sequence is formed based on each language token.

9. The multimodal ultrasound image recognition device as described in claim 8, characterized in that, The dynamic expert matching module is used to calculate the semantic matching degree between each token in the multimodal sequence and each preset dynamic expert based on the preset neural router and the multimodal embedding representation; Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined, including: Each token of the multimodal embedding representation is input into the neural router to extract the semantic representation of each token; Based on a preset similarity calculation algorithm, the semantic representation of each token is similar to the preset semantic representation of each dynamic expert to obtain the semantic matching degree between each token and each dynamic expert. Based on the semantic matching degree, several target dynamic experts and their corresponding participation weights are determined.

10. The multimodal ultrasound image recognition device as described in claim 9, characterized in that, The dynamic expert matching module is used to determine several target dynamic experts and their corresponding participation weights based on the semantic matching degree, including: The importance score of each dynamic expert is determined based on the semantic matching degree, and the dynamic experts are sorted in descending order based on the importance score to generate a dynamic expert list. Based on preset values, select several dynamic experts with the highest importance scores from the dynamic expert list as target dynamic experts, and calculate the corresponding participation weights based on the importance scores of the target dynamic experts.

Images