Lesion recognition method based on multi-modal ultrasonic time series data

Abstract

The present application relates to a kind of based on multimodal ultrasound time series data lesion identification method.It includes: obtaining the multimodal target time series data generated by ultrasound scanning to organ, the multimodal target time series data includes at least a group of multimodal data to be examined, multimodal target time series data is loaded to lesion instance segmentation model, to utilize the lesion instance segmentation model to carry out lesion segmentation identification processing to multimodal target time series data, and output the lesion instance state information corresponding to current multimodal target time series data;When it is determined that there is lesion in multimodal target time series data by lesion segmentation identification processing, then the lesion instance state information at least includes the category of each lesion.The present application can realize the continuous detection identification to lesion, improve the precision and reliability of lesion identification.

Description

Technical Field

[0001] This invention relates to a lesion identification method, and more particularly to a lesion identification method based on multimodal ultrasound time-series data. Background Technology

[0002] Currently, ultrasound diagnosis relies heavily on doctors' subjective observation and experience in interpreting ultrasound images. Differences may exist among doctors in image understanding, feature extraction, and the application of lesion grading standards, leading to low consistency in diagnostic results. Furthermore, the manual diagnostic process requires doctors to analyze ultrasound images frame by frame, which is time-consuming, labor-intensive, and demands a high level of professional expertise and clinical experience.

[0003] With the rapid development of artificial intelligence technology, deep learning algorithms have been widely used in the field of medical image analysis. Deep learning models can automatically learn and extract key features of lesions from a large number of ultrasound images of diseases, enabling automatic diagnosis and classification of lesions in various parts of the body. This improves the accuracy and consistency of grading, significantly increasing diagnostic efficiency, reducing the workload of doctors, and maintaining high result stability across different operators and equipment, thus reducing subjective errors.

[0004] Current deep learning algorithms for medical image analysis mainly utilize one mode of ultrasound images, such as B-mode ultrasound images, and do not apply to other modal data. This results in weak generalization ability, a tendency to miss diagnoses, and an inability to guarantee the continuity of lesion detection during the scanning process, making it difficult to meet the needs of ultrasound diagnosis in practical applications. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a lesion identification method based on multimodal ultrasound time-series data, which can realize continuous detection and identification of lesions and improve the accuracy and reliability of lesion identification.

[0006] According to the technical solution provided by the present invention, a lesion identification method based on multimodal ultrasound time-series data is provided, the lesion identification method comprising:

[0007] Acquire multimodal target time-series data generated from ultrasound scanning of organs, wherein the multimodal target time-series data includes at least one set of multimodal data to be examined, wherein...

[0008] When the multimodal target time series data includes multiple sets of multimodal test data, the multiple sets of multimodal test data are generated sequentially according to the time series of ultrasonic scanning.

[0009] Each group of multimodal data to be examined includes at least the B-mode image to be examined, the blood flow mode image to be examined, and the elastic mode image to be examined, which characterize the scanning state at the same time sequence.

[0010] Multimodal target time-series data is loaded into a lesion instance segmentation model to perform lesion segmentation and identification processing on the multimodal target time-series data, and outputs lesion instance status information corresponding to the current multimodal target time-series data.

[0011] During lesion segmentation and identification, multi-level feature extraction and cross-modal fusion feature extraction are performed sequentially on the multimodal data to be examined in each group to generate cross-modal fusion feature information corresponding to each group of multimodal data to be examined. Subsequently, the cross-modal fusion feature information of all groups of multimodal data to be examined is sequentially subjected to temporal feature fusion processing, multi-scale fusion processing and prediction output processing, and lesion instance status information is generated after prediction output processing.

[0012] When lesions are determined to exist in the multimodal target time series data through lesion segmentation and identification processing, the lesion instance status information shall include at least the category of each lesion.

[0013] When two temporally correlated multimodal target time series data exist, and lesion segmentation and identification processing determines that lesions exist within each multimodal target time series data, then lesion tracking is performed on the lesions identified in the two temporally correlated multimodal target data.

[0014] Lesion tracking includes:

[0015] For each multimodal target time series data, obtain the lesion detection box for each lesion after lesion segmentation and identification processing;

[0016] The lesion detection box of each lesion in the previous time series multimodal target time series data is used as the basic tracking box, and the image region corresponding to the basic tracking box is configured as the basic tracking region. At the same time, the lesion detection box of each lesion in the next time series multimodal target time series data is used as the detection box to be matched, and the image region corresponding to each detection box to be matched is configured as the region to be matched.

[0017] Motion prediction is performed on the basic tracking boxes to generate basic tracking prediction boxes. Then, Hungarian matching detection is performed between the basic tracking prediction boxes and each target detection box.

[0018] When the basic tracking prediction box and the target detection box satisfy the Hungarian matching, the basic tracking prediction box is configured as the tracking target box, and the corresponding target detection box is configured as the tracking matching box.

[0019] When the tracking target box does not meet the tracking matching box that satisfies the Hungarian matching, the image region corresponding to the tracking target box is subjected to foreground target supplementation segmentation on the main modality map of the current frame group of multimodal data to be inspected in the subsequent time series multimodal data, so as to determine the lesion information of the image region corresponding to the tracking target box after foreground target supplementation segmentation.

[0020] When performing Hungarian matching detection between the basic tracking region and the region to be matched, the following is included:

[0021] The basic tracking bounding boxes are preprocessed using Hungarian matching to generate basic tracking prediction bounding boxes.

[0022] The Hungarian matching preprocessing includes appearance feature extraction, smoothing, and / or Kalman filtering motion prediction.

[0023] Based on the appearance and area of ​​the basic tracking prediction box and the corresponding detection box to be matched, the cost matrix is ​​calculated.

[0024] Based on the calculated cost matrix, the Hungarian matching algorithm is used to solve the Hungarian matching problem to determine the Hungarian matching status between the basic tracking region and the corresponding preprocessing region to be matched.

[0025] When performing tracking and matching judgments, the following are included:

[0026] Calculate the Intersection over Union (IOU) between the target bounding box and the tracking matching bounding boxes that satisfy the Hungarian matching.

[0027] When the calculated Intersection over Union (IOU) meets the tracking matching threshold, the tracking target box and the tracking matching box that meets the Hungarian matching are considered to be matched. Otherwise, the tracking target box and the tracking matching box that meets the Hungarian matching are not considered to be matched.

[0028] When the presence of a lesion is determined in the multimodal target time series data through lesion segmentation and identification processing, the lesion instance status information is marked on the main modality map of the current frame group of multimodal data to be inspected in the current multimodal target time series data;

[0029] Based on the main modality map of the multimodal data to be detected in the frame group and the lesion detection box marked on the main modality map for each lesion, lesion feature description processing is performed on each lesion to determine the lesion region features of each lesion. The lesion region features include echo type, shape and / or edge features.

[0030] The generated lesion instance status information is loaded into the constructed diagnostic report generation model, so that the corresponding diagnostic report can be generated using the diagnostic report generation model.

[0031] When building a diagnostic report generation model, the following are included:

[0032] A large language basic model is provided, and the large language basic model is fine-tuned using the constructed large language model training dataset. During fine-tuning, the QLoRA method is used.

[0033] After training the basic large language model to the target state using the large language model training dataset, a diagnostic report generation model is formed based on the trained basic large language model.

[0034] When generating a diagnostic report, the diagnostic report generation model processes the input lesion instance status information and combines it with retrieval enhancement to generate the corresponding diagnostic report.

[0035] When training the large language basic model, the speech model data samples in the large language model training dataset are first segmented and merged using a word segmenter to form speech model training samples.

[0036] When fine-tuning using the QLoRA method, the pre-trained weights of the large language base model are frozen, and learnable low-rank matrices are inserted only into the key layers of the large language base model. and low-rank matrices The query matrix and value matrix are configured as a multi-head attention mechanism within the basic model of a large language.

[0037] The lesion instance segmentation model comprises a backbone network, a cross-modal attention module, a temporal feature fusion module, a neck network, and a segmentation output head, connected sequentially.

[0038] During lesion segmentation and identification, a backbone network is used to perform multi-level feature extraction on each group of multimodal data to form at least a shallow feature map group, a medium-level feature map group, and a deep feature map group.

[0039] The shallow feature map group includes shallow B-mode feature map, shallow blood flow mode feature map, and shallow elasticity mode feature map;

[0040] The mid-layer feature map group includes the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elasticity mode feature map;

[0041] The deep feature map group includes deep B-mode feature map, deep blood flow mode feature map, and deep elasticity mode feature map;

[0042] Cross-modal fusion feature extraction is performed using a cross-modal attention module to perform cross-modal fusion on shallow feature map groups, mid-level feature map groups, and deep feature map groups respectively, and shallow cross-modal fusion feature maps, mid-level cross-modal fusion feature maps, and deep cross-modal fusion feature maps are generated respectively. Cross-modal fusion feature information of the current group of multimodal data to be detected is formed based on the shallow cross-modal fusion feature maps, mid-level cross-modal fusion feature maps, and deep cross-modal fusion feature maps.

[0043] The temporal feature fusion module is used to perform temporal feature fusion processing on the cross-modal fused feature information corresponding to the temporal data of all multimodal targets.

[0044] During the temporal feature fusion process, all shallow cross-modal fusion feature maps are fused in a temporal sequence to generate shallow temporal fusion features, all mid-level cross-modal fusion feature maps are fused in a temporal sequence to generate mid-level temporal fusion features, and all deep cross-modal fusion feature maps are fused in a temporal sequence to generate deep temporal fusion features.

[0045] The neck network is used to perform multi-scale fusion of shallow, mid-level and deep temporal fusion features. After multi-scale fusion, the feature is processed by the segmentation output head to generate the corresponding lesion instance status information.

[0046] The backbone network comprises a shallow shared layer, a shallow backbone layer, a middle backbone layer, and a deep backbone layer connected in sequence.

[0047] Each of the shallow, middle, and deep main layers includes three backbone branches, and the corresponding three backbone branches in the shallow, middle, and deep main layers are connected to form three modal feature extraction branches.

[0048] When performing multi-level feature extraction processing on the multimodal data to be tested in each group, the B-mode image, blood flow mode image, and elasticity mode image to be tested are processed through a shallow shared layer, and then processed through the corresponding modal feature extraction branch.

[0049] The superficial B-mode feature map, superficial blood flow mode feature map, and superficial elasticity mode feature map are output from the corresponding skeletal branches of the superficial main trunk layer, respectively.

[0050] The skeletal branches corresponding to the main mid-layer layer output the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elasticity mode feature map, respectively.

[0051] The deep B-mode feature map, deep blood flow mode feature map, and deep elastic mode feature map are output from the corresponding skeletal branches of the deep main trunk layer, respectively.

[0052] The shallow shared layer includes at least two cascaded shallow shared convolutional modules, wherein,

[0053] Two shallow shared convolutional modules serve as the input and output layers of their respective shallow shared layers;

[0054] Each shallow shared convolutional module includes a two-dimensional convolutional layer, a batch normalization layer, and a shallow shared activation function layer connected in sequence.

[0055] The shallow shared activation function layer uses the SiLU activation function.

[0056] Each backbone branch of the shallow backbone layer includes a shallow first convolutional-attention fusion module, a shallow convolutional layer, and a shallow second convolutional-attention fusion module connected in sequence.

[0057] Each backbone branch within the middle layer includes a middle-layer spatial-channel decoupled downsampling module and a middle-layer convolution-attention fusion module connected in sequence.

[0058] Each backbone branch within the deep backbone layer includes, in sequence, a deep spatial-channel decoupling downsampling module, a deep C2fCIB module, a feature pyramid pooling module, and a PSA module.

[0059] The shallow first convolutional-attention fusion module is adapted and connected to the shallow shared layer, and the shallow second convolutional-attention fusion module is adapted and connected to the middle spatial-channel decoupled downsampling module. The shallow second convolutional-attention fusion module, which passes through the three backbone branches in the shallow main layer, outputs the shallow B-mode feature map, the shallow blood flow mode feature map, and the shallow elastic mode feature map, respectively.

[0060] The mid-layer convolution-attention fusion module is adapted and connected to the deep spatial-channel decoupled downsampling module, and the mid-layer convolution-attention fusion module through the three backbone branches in the mid-layer backbone layer outputs the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elastic mode feature map, respectively.

[0061] The PSA modules of the three backbone branches within the deep main layer output deep B-mode feature maps, deep blood flow mode feature maps, and deep elastic mode feature maps, respectively.

[0062] The shallow first convolutional-attention fusion module, the shallow second convolutional-attention fusion module, and the middle convolutional-attention fusion module all adopt the same structural form, including cross-stage partial fusion of the first convolutional block, wherein...

[0063] The output of the cross-stage partial fusion first convolutional block is connected to one input of the cross-stage first convolutional self-attention module, the cross-stage second convolutional self-attention module, and the cross-stage partial fusion splicer, respectively.

[0064] The first self-attention module and the second self-attention module of the cross-stage are respectively connected to the corresponding input end of the cross-stage partial fusion splicer. The output end of the cross-stage partial fusion splicer is connected to the second convolutional block of the cross-stage fusion, and the second convolutional block of the cross-stage fusion serves as the output layer.

[0065] The cross-modal attention module includes three cross-modal attention units, which are used to perform cross-modal fusion on shallow feature map groups, mid-level feature map groups, and deep feature map groups, respectively.

[0066] When performing cross-modal fusion, the corresponding B-mode feature map is used as the master mode, and the corresponding blood flow mode feature map and elasticity mode feature map are guided by the cross-modal attention mechanism based on the Transformer structure.

[0067] When the cross-modal attention mechanism based on the Transformer structure guides the corresponding blood flow modal feature map and elastic modal feature map, the corresponding B modal feature map is used as the query source, and the corresponding blood flow modal feature map and elastic modal feature map are used as the key source and value source, respectively.

[0068] The temporal feature fusion module includes three temporal feature fusion units, which are used to perform temporal fusion on all shallow cross-modal fusion feature maps, mid-level cross-modal fusion feature maps, and deep cross-modal fusion feature maps respectively.

[0069] During temporal fusion, a global instance query is determined on each cross-modal fusion feature map, and each global instance query is decomposed into frame-level features using a dynamic attention mechanism;

[0070] All corresponding frame-level features of the same type of cross-modal fusion feature map are concatenated to form a corresponding frame-by-frame concatenated query.

[0071] Each frame-by-frame concatenation query is weighted and aggregated to form the corresponding temporal fusion feature.

[0072] When generating multimodal test data for each group, the following steps are included:

[0073] The source images of mode B, blood flow, and elasticity were acquired. Subsequently, image preprocessing was performed on each of these source images to generate corresponding target mode B, blood flow, and elasticity images.

[0074] When performing image preprocessing on the B-mode source image, at least wavelet threshold filtering processing is performed on the B-mode source image;

[0075] When performing image preprocessing on blood flow modality source images, the blood flow modality source images are first converted to the HSV color space, and then the saturation channel features are enhanced in the HSV color space before being converted back to RGB images.

[0076] When performing image preprocessing on the source image of the elastic modality, at least the pseudo-color image should be mapped and converted into a single-channel grayscale image.

[0077] Advantages of this invention: During lesion identification, multimodal target time-series data is acquired. The multimodal target time-series data includes several sets of multimodal data to be examined, including B-mode images to be examined, blood flow modal images to be examined, and elastic modal images to be examined. When using a lesion instance segmentation model for skill-based lesion segmentation and identification based on one or more sets of multimodal data to be examined within the multimodal target time-series data, the lesion detection capability of ultrasound multimodal data can be improved. Lesion information can be obtained from different angles, making up for the limitations of single modality in terms of resolution, tissue contrast, etc. It can capture the changes of lesions over time and reduce diagnostic errors caused by the limited information content of single frames and single modal images.

[0078] When there is temporally correlated multimodal target time-series data, lesions identified through lesion segmentation and identification are tracked to avoid missing lesions in organs; in addition, the lesion region characteristics of each lesion can be determined through lesion feature description processing.

[0079] By utilizing a large language model to construct a diagnostic report generation model, the model possesses powerful language understanding and analysis capabilities. It integrates information from multiple sources to conduct a comprehensive and in-depth analysis of lesions, providing more scientific and reasonable diagnostic results. Ultimately, it outputs a structured report, making the diagnostic results clearer and more standardized, easier for clinicians to understand, improving the efficiency of medical diagnosis, and facilitating subsequent medical record and data management. Attached Figure Description

[0080] Figure 1 This is a schematic flowchart of one embodiment of the lesion identification method of the present invention.

[0081] Figure 2 This is a structural block diagram of one embodiment of the lesion instance segmentation model of the present invention.

[0082] Figure 3 This is a schematic diagram of an embodiment of the cooperation between the backbone network and the cross-modal attention module of the present invention.

[0083] Figure 4 This is a schematic diagram of one embodiment of the temporal feature fusion module, neck network, and segmentation output head of the present invention.

[0084] Figure 5 This is a schematic diagram of an embodiment of the shallow first convolution-attention fusion module of the present invention. Detailed Implementation

[0085] The present invention will be further described below with reference to specific accompanying drawings and embodiments.

[0086] To achieve continuous detection and identification of lesions and improve the accuracy and reliability of lesion identification, this invention provides a lesion identification method based on multimodal ultrasound time-series data. Specifically, the lesion identification method includes:

[0087] Acquire multimodal target time-series data generated from ultrasound scanning of organs, wherein the multimodal target time-series data includes at least one set of multimodal data to be examined, wherein...

[0088] When the multimodal target time series data includes multiple sets of multimodal test data, the multiple sets of multimodal test data are generated sequentially according to the time series of ultrasonic scanning.

[0089] Each group of multimodal data to be examined includes at least the B-mode image to be examined, the blood flow mode image to be examined, and the elastic mode image to be examined, which characterize the scanning state at the same time sequence.

[0090] Multimodal target time-series data is loaded into a lesion instance segmentation model to perform lesion segmentation and identification processing on the multimodal target time-series data, and outputs lesion instance status information corresponding to the current multimodal target time-series data.

[0091] During lesion segmentation and identification, multi-level feature extraction and cross-modal fusion feature extraction are performed sequentially on the multimodal data to be examined in each group to generate cross-modal fusion feature information corresponding to each group of multimodal data to be examined. Subsequently, the cross-modal fusion feature information of all groups of multimodal data to be examined is sequentially subjected to temporal feature fusion processing, multi-scale fusion processing and prediction output processing, and lesion instance status information is generated after prediction output processing.

[0092] When lesions are determined to exist in the multimodal target time series data through lesion segmentation and identification processing, the lesion instance status information shall include at least the category of each lesion.

[0093] It should be noted that the lesion in this invention specifically refers to a lesion appearing within a human organ, which may include the breast, thyroid gland, liver, carotid artery, gallbladder, etc. The type of organ can be selected and determined as needed, and will not be listed here. The lesion specifically refers to the area / location of the lesion on the organ. When identifying lesions, the organ should be scanned with ultrasound. Multimodal time-series data specifically means that lesion identification should utilize ultrasound data from multiple modalities, and there should be a temporal relationship between the data. The temporal relationship generally refers to the time sequence generated by the ultrasound scan of the modal data.

[0094] Figure 1The diagram illustrates an embodiment of the present invention for lesion identification. As shown, lesion identification involves performing an ultrasound scan of the organ and acquiring multimodal target time-series data after the ultrasound scan. In other words, the present invention utilizes multimodal target time-series data to identify lesions in organs. Specifically, the multimodal target time-series data should include at least one set of multimodal data to be examined. Of course, the multimodal target time-series data can also include multiple sets of multimodal data to be examined, and the number of multimodal data to be examined can be selected as needed.

[0095] When multimodal target time-series data includes multiple sets of multimodal data to be examined, these sets of data should be sequentially formed according to the time sequence of the ultrasound scan. In other words, adjacent sets of data should have a temporal relationship. Generally, when an ultrasound probe is used to scan an organ, a corresponding ultrasound video is generated. A sliding window is then used on the generated ultrasound video to select and form multimodal target time-series data.

[0096] It should be understood that the window size and step size of the sliding window can be configured to determine the corresponding multimodal target time series data. For example, if the window size is 5 frames and the sliding window step size is 1 frame, then the multimodal target time series data includes 5 sets of multimodal data to be inspected. For instance, the multimodal data to be inspected corresponding to time series 1 to 5 can be formed into one multimodal target time series data, while the multimodal data to be inspected corresponding to time series 2 to 6 can be formed into another multimodal target time series data. Other cases can be referred to the explanation here, and will not be illustrated one by one. In addition, when there is only one set of multimodal data to be inspected in the multimodal target time series data, the window size of the sliding window is 1 frame, and of course, the sliding window step size can still be 1.

[0097] Each set of multimodal test data should include at least the B-mode image, the blood flow mode image, and the elasticity mode image. Furthermore, the B-mode image, blood flow mode image, and elasticity mode image within each set of multimodal test data should represent the scanning state at the same time sequence; that is, the B-mode image, blood flow mode image, and elasticity mode image should be generated synchronously. In specific implementation, the ultrasound probe performs an ultrasound scan of the organ in one mode and generates an image of the corresponding mode. Subsequently, by using the transformation relationship of the corresponding modes, the image of the corresponding mode can be obtained. For example, if the ultrasound probe is configured to perform B-mode imaging of the organ, the B-mode image to be tested can be generated. Then, based on the transformation relationship between the blood flow mode, elasticity mode, and B-mode, the corresponding blood flow mode image and elasticity mode image to be tested can be generated after appropriate transformation processing. Other cases can be referred to here, and will not be illustrated further. In addition, the conversion relationship between different modal images can be consistent with existing technologies, and will not be elaborated here.

[0098] To achieve automated lesion identification, a lesion instance segmentation model should be constructed. Figure 1 It is known that during lesion identification, multimodal target time series data should be loaded into the lesion instance segmentation model so that the lesion instance segmentation model can be used to segment and identify lesions in the multimodal target time series data, and corresponding lesion instance status information can be generated. The lesion instance status information is the information describing the lesion situation in the current multimodal target time series data. The lesion instance status information can be referred to in the following description.

[0099] Since multimodal target time-series data may include multiple sets of multimodal data to be examined, when performing lesion segmentation and identification processing, each set of multimodal data to be examined should first undergo multi-level feature extraction processing and cross-modal fusion feature extraction processing sequentially. After cross-modal fusion feature extraction, cross-modal fusion feature information corresponding to each set of data to be examined can be generated. Therefore, when there are multiple sets of multimodal data to be examined, multiple sets of cross-modal fusion feature information can be obtained. Subsequently, the cross-modal fusion feature information of all sets of multimodal data to be examined is sequentially subjected to time-series feature fusion processing, multi-scale fusion processing, and prediction output processing, and lesion instance status information is generated after prediction output processing.

[0100] It is understandable that lesions may or may not exist within an organ. Therefore, when the presence of a lesion is determined through lesion segmentation and identification, the lesion instance status information can at least include the lesion category, which represents the type of abnormality occurring in the organ. It should be noted that the lesion category is related to the training samples used to construct the lesion instance segmentation model. That is, determining the lesion category should be based on a lesion category labeled in the training samples used to construct the lesion instance segmentation model. The details of the training samples used to construct the lesion instance segmentation model will be explained below.

[0101] Since B-mode images, blood flow modal images, and elasticity modal images can characterize the scanning state of organs from different perspectives, this invention can improve the accuracy and reliability of lesion identification when using multimodal target time-series data for lesion segmentation and identification. As can be seen from the above process of forming multimodal target data, this invention can achieve continuous detection and identification.

[0102] In one embodiment of the present invention, generating multimodal test data for each group includes:

[0103] The source images of mode B, blood flow, and elasticity were acquired. Subsequently, image preprocessing was performed on each of these source images to generate corresponding target mode B, blood flow, and elasticity images.

[0104] When performing image preprocessing on the B-mode source image, at least wavelet threshold filtering processing is performed on the B-mode source image;

[0105] When performing image preprocessing on blood flow modality source images, the blood flow modality source images are first converted to the HSV color space, and then the saturation channel features are enhanced in the HSV color space before being converted back to RGB images.

[0106] When performing image preprocessing on the source image of the elastic modality, at least the pseudo-color image should be mapped and converted into a single-channel grayscale image.

[0107] To further improve the accuracy of lesion identification, when generating multimodal data to be examined, B-mode source images, blood flow modal source images, and elastic modal source images are generally generated first. Then, corresponding image preprocessing is performed so that the B-mode image, blood flow modal image, and elastic modal image to be examined can be generated after image preprocessing. The generation methods of B-mode source images, blood flow modal source images, and elastic modal source images can be referred to the above-mentioned descriptions.

[0108] In practice, image preprocessing of the modal source image typically involves wavelet threshold filtering. Wavelet threshold filtering effectively suppresses noise while preserving lesion edge information. Since the blood flow modal source image is a color Doppler blood flow image, image preprocessing can involve converting it to the HSV color space, enhancing the saturation channel features in the HSV color space, and then converting it back to RGB. The method for converting the RGB blood flow modal source image to the HSV color space is consistent with existing techniques. Furthermore, the method for enhancing the saturation channel features is also consistent with existing technologies; specific examples will not be provided here.

[0109] In addition, when performing image preprocessing on the elastic modality source image, at least the pseudo-color image should be mapped to a single-channel grayscale image. The specific method of mapping the elastic modality source image from a pseudo-color image to a single-channel grayscale image can be consistent with existing technologies, and will not be elaborated here.

[0110] In one embodiment of the present invention, the lesion instance segmentation model includes a backbone network, a cross-modal attention module, a temporal feature fusion module, a neck network, and a segmentation output head connected in sequence, wherein,

[0111] During lesion segmentation and identification, a backbone network is used to perform multi-level feature extraction on each group of multimodal data to form at least a shallow feature map group, a medium-level feature map group, and a deep feature map group.

[0112] The shallow feature map group includes shallow B-mode feature map, shallow blood flow mode feature map, and shallow elasticity mode feature map;

[0113] The mid-layer feature map group includes the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elasticity mode feature map;

[0114] The deep feature map group includes deep B-mode feature map, deep blood flow mode feature map, and deep elasticity mode feature map;

[0115] Cross-modal fusion feature extraction is performed using a cross-modal attention module to perform cross-modal fusion on shallow feature map groups, mid-level feature map groups, and deep feature map groups respectively, and shallow cross-modal fusion feature maps, mid-level cross-modal fusion feature maps, and deep cross-modal fusion feature maps are generated respectively. Cross-modal fusion feature information of the current group of multimodal data to be detected is formed based on the shallow cross-modal fusion feature maps, mid-level cross-modal fusion feature maps, and deep cross-modal fusion feature maps.

[0116] The temporal feature fusion module is used to perform temporal feature fusion processing on the cross-modal fused feature information corresponding to the temporal data of all multimodal targets.

[0117] During the temporal feature fusion process, all shallow cross-modal fusion feature maps are fused in a temporal sequence to generate shallow temporal fusion features, all mid-level cross-modal fusion feature maps are fused in a temporal sequence to generate mid-level temporal fusion features, and all deep cross-modal fusion feature maps are fused in a temporal sequence to generate deep temporal fusion features.

[0118] The neck network is used to perform multi-scale fusion of shallow, mid-level and deep temporal fusion features. After multi-scale fusion, the feature is processed by the segmentation output head to generate the corresponding lesion instance status information.

[0119] Figure 2 The figure illustrates an embodiment of a lesion instance segmentation model. As shown in the figure, the lesion instance segmentation model may include a backbone network, a cross-modal attention module, a temporal feature fusion module, a neck network, and a segmentation output head connected in sequence. During lesion segmentation and recognition processing, the backbone network receives each group of multimodal data to be examined and performs multi-level feature extraction processing on the multimodal data to be examined. Specifically, the multi-level feature extraction processing performed by the backbone network of the present invention can be for shallow, middle, and deep feature extraction, thereby forming shallow feature map groups, middle feature map groups, and deep feature map groups.

[0120] Specifically, when the backbone network performs multi-level feature extraction, it should simultaneously extract features from the B-mode image, blood flow mode image, and elasticity mode image that belong to the same multi-modal test data. Thus, the shallow feature map group includes the shallow B-mode feature map, the shallow blood flow mode feature map, and the shallow elasticity mode feature map; similarly, the middle feature map group includes the middle B-mode feature map, the middle blood flow mode feature map, and the middle elasticity mode feature map; and the deep feature map group includes the deep B-mode feature map, the deep blood flow mode feature map, and the deep elasticity mode feature map.

[0121] In practice, a cross-modal attention module is used for cross-modal fusion feature extraction. Specifically, cross-modal fusion of the shallow B-mode feature map, shallow blood flow modal feature map, and shallow elasticity modal feature map within the shallow feature map group yields a shallow cross-modal fusion feature map. Similarly, cross-modal fusion feature extraction of the mid-level feature map group generates a mid-level cross-modal fusion feature map, and cross-modal fusion feature extraction of the deep feature map group generates a deep cross-modal fusion feature map. It is understood that for each group of multimodal data to be examined, the cross-modal fusion feature information should include shallow, mid-level, and deep cross-modal fusion feature maps.

[0122] The temporal feature fusion module is used to perform temporal feature fusion processing on the cross-modal fusion feature information corresponding to all multimodal target temporal data. As mentioned above, if the multimodal target temporal data includes 5 sets of multimodal data to be inspected, then the cross-modal fusion feature information corresponding to the 5 sets of multimodal data to be inspected should be processed using temporal feature fusion. When the number of sets of multimodal data to be inspected is other, please refer to the explanation here. Specifically, during the temporal feature fusion processing, all shallow cross-modal fusion feature maps are time-series fused to generate shallow temporal fusion features, all mid-level cross-modal fusion feature maps are time-series fused to generate mid-level temporal fusion features, and all deep cross-modal fusion feature maps are time-series fused to generate deep temporal fusion features.

[0123] In addition, within the lesion instance segmentation model, the neck network is used to perform multi-scale fusion of shallow temporal fusion features, mid-level temporal fusion features, and deep temporal fusion features. After multi-scale fusion, the segmentation output head is used for prediction output processing to generate the corresponding lesion instance state information.

[0124] In one embodiment of the present invention, the backbone network includes a shallow shared layer, a shallow backbone layer, a middle backbone layer, and a deep backbone layer connected in sequence, wherein,

[0125] Each of the shallow, middle, and deep main layers includes three backbone branches, and the corresponding three backbone branches in the shallow, middle, and deep main layers are connected to form three modal feature extraction branches.

[0126] When performing multi-level feature extraction processing on the multimodal data to be tested in each group, the B-mode image, blood flow mode image, and elasticity mode image to be tested are processed through a shallow shared layer, and then processed through the corresponding modal feature extraction branch.

[0127] The superficial B-mode feature map, superficial blood flow mode feature map, and superficial elasticity mode feature map are output from the corresponding skeletal branches of the superficial main trunk layer, respectively.

[0128] The skeletal branches corresponding to the main mid-layer layer output the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elasticity mode feature map, respectively.

[0129] The deep B-mode feature map, deep blood flow mode feature map, and deep elastic mode feature map are output from the corresponding skeletal branches of the deep main trunk layer, respectively.

[0130] It should be noted that the lesion instance segmentation model of this invention is based on the YOLOv10 model. Unlike existing YOLOv10 models, the backbone network of this invention may include shallow shared layers, shallow backbone layers, mid-level backbone layers, and deep backbone layers. The shallow shared layers are used in the initial feature extraction stage, that is, to extract lower-level features, such as low-level edges, textures, and local shape structure features in the feature map. Compared with the completely shared approach in existing YOLOv10 models, the use of shallow shared layers emphasizes distinguishing the unique features of each modality image. Compared with each modality having its own independent backbone, this reduces computational overhead without sacrificing accuracy. The following section will combine... Figure 3 The backbone network of this invention will be described in detail below.

[0131] In one embodiment of the present invention, the shallow shared layer includes at least two cascaded shallow shared convolutional modules, wherein,

[0132] Two shallow shared convolutional modules serve as the input and output layers of their respective shallow shared layers;

[0133] Each shallow shared convolutional module includes a two-dimensional convolutional layer, a batch normalization layer, and a shallow shared activation function layer connected in sequence.

[0134] The shallow shared activation function layer uses the SiLU activation function.

[0135] Figure 3In this model, convolutional modules Conv1 and Conv2 can each form two cascaded shallow shared convolutional modules. Conv1 forms stage 1 within the backbone network, and Conv2 forms stage 2. Conv1 serves as the input layer of its respective shallow shared layer, receiving the B-mode image, blood flow mode image, and elasticity mode image. Conv2 connects to the corresponding shallow backbone layer. Since the SiLU activation function is continuously differentiable within its domain, setting the shallow shared activation function layer to SiLU results in smoother gradients, reducing the risk of gradient explosion or vanishing gradients and facilitating stable convergence during training.

[0136] In one embodiment of the present invention, each backbone branch of the shallow backbone layer includes a shallow first convolutional-attention fusion module, a shallow convolutional layer, and a shallow second convolutional-attention fusion module connected in sequence.

[0137] Each backbone branch within the middle layer includes a middle-layer spatial-channel decoupled downsampling module and a middle-layer convolution-attention fusion module connected in sequence.

[0138] Each backbone branch within the deep backbone layer includes, in sequence, a deep spatial-channel decoupling downsampling module, a deep C2fCIB module, a feature pyramid pooling module, and a PSA module.

[0139] The shallow first convolutional-attention fusion module is adapted and connected to the shallow shared layer, and the shallow second convolutional-attention fusion module is adapted and connected to the middle spatial-channel decoupled downsampling module. The shallow second convolutional-attention fusion module, which passes through the three backbone branches in the shallow main layer, outputs the shallow B-mode feature map, the shallow blood flow mode feature map, and the shallow elastic mode feature map, respectively.

[0140] The mid-layer convolution-attention fusion module is adapted and connected to the deep spatial-channel decoupled downsampling module, and the mid-layer convolution-attention fusion module through the three backbone branches in the mid-layer backbone layer outputs the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elastic mode feature map, respectively.

[0141] The PSA modules of the three backbone branches within the deep main layer output deep B-mode feature maps, deep blood flow mode feature maps, and deep elastic mode feature maps, respectively.

[0142] Understandably, when setting up a shallow shared layer within the backbone network and performing multi-level feature extraction, the shallow shared layer is first used to process the B-mode image, blood flow mode image, and elasticity mode image to be examined. Subsequently, feature extraction is performed independently and non-shared through the shallow backbone layer, middle backbone layer, and deep backbone layer. Therefore, each of the shallow backbone layer, middle backbone layer, and deep backbone layer includes three backbone branches, and the corresponding three backbone branches in the shallow backbone layer, middle backbone layer, and deep backbone layer are connected to form three modal feature extraction branches. Subsequently, the three modal feature extraction branches can be used to extract features from the B-mode, blood flow mode, and elasticity mode respectively.

[0143] During operation, the three modal feature extraction branches are assigned to the B-mode, blood flow modality, and elasticity modality. The feature maps processed by the shallow shared layer are then assigned to their corresponding modal feature extraction branches. For example, for the B-mode image to be examined, after processing by the shallow shared layer, it will always be configured to the corresponding modal feature extraction branch. Similarly, the corresponding multi-level feature extraction paths for the blood flow modality image and the elasticity modality image to be examined can be determined. In specific implementation, after configuring the corresponding multi-level feature extraction paths, the images can be processed by the shallow shared layer according to their modal type, and then assigned to the corresponding multi-level extraction paths using techniques commonly used in this technical field.

[0144] In specific implementation, the three backbone branches in the shallow backbone layer adopt the same form. In one embodiment of the present invention, each backbone branch of the shallow backbone layer includes a shallow first convolution-attention fusion module, a shallow convolutional layer, and a shallow second convolution-attention fusion module connected in sequence.

[0145] Each backbone branch within the middle layer includes a middle-layer spatial-channel decoupled downsampling module and a middle-layer convolution-attention fusion module connected in sequence.

[0146] Each backbone branch within the deep backbone layer includes, in sequence, a deep spatial-channel decoupling downsampling module, a deep C2fCIB module, a feature pyramid pooling module, and a PSA module.

[0147] The shallow first convolutional-attention fusion module is adapted and connected to the shallow shared layer, and the shallow second convolutional-attention fusion module is adapted and connected to the middle spatial-channel decoupled downsampling module. The shallow second convolutional-attention fusion module, which passes through the three backbone branches in the shallow main layer, outputs the shallow B-mode feature map, the shallow blood flow mode feature map, and the shallow elastic mode feature map, respectively.

[0148] The mid-layer convolution-attention fusion module is adapted and connected to the deep spatial-channel decoupled downsampling module, and the mid-layer convolution-attention fusion module through the three backbone branches in the mid-layer backbone layer outputs the mid-layer B-mode feature map, the mid-layer blood flow mode feature map, and the mid-layer elastic mode feature map, respectively.

[0149] The PSA modules of the three backbone branches within the deep main layer output deep B-mode feature maps, deep blood flow mode feature maps, and deep elastic mode feature maps, respectively.

[0150] Figure 3 In this context, stage 3 refers to the shallow main layer; for the shallow main layer, along... Figure 3 From left to right, C2f-ConvFormer1 is the first shallow convolutional-attention fusion module in the first backbone branch, Conv3 is the shallow convolutional layer in the first backbone branch, and C2f-ConvFormer4 is the first shallow convolutional-attention fusion module in the first backbone branch. Similarly, C2f-ConvFormer2 is the first shallow convolutional-attention fusion module in the second backbone branch, Conv4 is the shallow convolutional layer in the second backbone branch, C2f-ConvFormer5 is the second shallow convolutional-attention fusion module in the second backbone branch, C2f-ConvFormer3 is the first shallow convolutional-attention fusion module in the third backbone branch, Conv5 is the shallow convolutional layer in the third backbone branch, and C2f-ConvFormer6 is the second shallow convolutional-attention fusion module in the third backbone branch.

[0151] Figure 3 In this context, stage 4 is the middle backbone layer; for the middle backbone layer, along... Figure 3 From left to right, SCDown1 is the mid-level spatial-channel decoupled downsampling module in the first backbone branch, C2f-ConvFormer7 is the mid-level convolution-attention fusion module in the first backbone branch, and similarly, SCDown2 is the mid-level spatial-channel decoupled downsampling module in the second backbone branch, C2f-ConvFormer8 is the mid-level convolution-attention fusion module in the second backbone branch, SCDown3 is the mid-level spatial-channel decoupled downsampling module in the third backbone branch, and C2f-ConvFormer9 is the mid-level convolution-attention fusion module in the third backbone branch.

[0152] Figure 3 In this context, stage 5 is the deep backbone layer. For the deep backbone layer, along... Figure 3From left to right, SCDown4 is the deep spatial-channel decoupled downsampling module in the first backbone branch, C2fCIB1 is the deep C2fCIB module in the first backbone branch, SPPF1 is the feature pyramid pooling module in the first backbone branch, and PSA1 is the PSA module in the first backbone branch. Similarly, SCDown5 is the deep spatial-channel decoupled downsampling module in the second backbone branch, C2fCIB2 is the deep C2fCIB module in the second backbone branch, SPPF2 is the feature pyramid pooling module in the second backbone branch, and PSA2 is the PSA module in the second backbone branch; SCDown6 is the deep spatial-channel decoupled downsampling module in the third backbone branch, C2fCIB3 is the deep C2fCIB module in the third backbone branch, SPPF3 is the feature pyramid pooling module in the third backbone branch, and PSA3 is the PSA module in the third backbone branch.

[0153] In one embodiment of the present invention, the shallow first convolutional-attention fusion module, the shallow second convolutional-attention fusion module, and the middle convolutional-attention fusion module adopt the same structural form, including a cross-stage partial fusion of the first convolutional block, wherein,

[0154] The output of the cross-stage partial fusion first convolutional block is connected to one input of the cross-stage first convolutional self-attention module, the cross-stage second convolutional self-attention module, and the cross-stage partial fusion splicer, respectively.

[0155] The first self-attention module and the second self-attention module of the cross-stage are respectively connected to the corresponding input end of the cross-stage partial fusion splicer. The output end of the cross-stage partial fusion splicer is connected to the second convolutional block of the cross-stage fusion, and the second convolutional block of the cross-stage fusion serves as the output layer.

[0156] It should be noted that, Figure 3 and Figure 4 In this model, all C2f-ConvFormers have the same form. When forming a C2f-ConvFormer, the main approach is to replace the Bottleneck module within the C2f module in the existing YOLOv10 model with the ConvFormer module. Figure 5 An embodiment of C2f-ConvFormer is shown in the figure. Figure 5 In this context, Conv Block1 is the first convolutional block for cross-stage partial fusion. This first convolutional block can be a 1x1 convolution. Using this first convolutional block, the feature map input to the C2f-ConvFormer can be split into two branches.

[0157] In the first branch of the split, the feature map is processed by the first convolutional self-attention module and the second self-attention module across the stage. In the second branch of the split, the feature map is not processed and is directly loaded into the cross-stage partial fusion stitcher.

[0158] Figure 5 In this diagram, ConvFormer Block1 is the first convolutional self-attention module across stages, ConvFormer Block2 is the second convolutional self-attention module across stages, Concat5 is the cross-stage partial fusion stitcher, and Conv Block2 is the second convolutional block for cross-stage fusion. The output of the cross-stage partial fusion stitcher is connected to the second convolutional block for cross-stage fusion. The second convolutional block for cross-stage fusion is generally also a 1*1 convolutional layer, and the original number of channels in the input feature map can be restored through the second convolutional block for cross-stage fusion. It should be noted that the cross-stage partial fusion stitcher is used to implement channel stitching, and all other stitchers mentioned in this invention are also channel stitching.

[0159] It should be noted that the depthwise separable convolution and standard convolution in the ConvFormer module aggregate local spatial information, which can effectively extract local correlations in the image and avoid the computational burden caused by the pure self-attention mechanism. At the same time, a global information interaction mechanism is introduced, using a channel multilayer perceptron to focus on the interaction and transformation of features in each channel of the feature map, and a large number of data dimensionality reduction convolutional layers with a kernel size of 1*1 are used. While improving the model's ability to understand complex graph phenomena, the number of parameters is reduced, maintaining the lightweight nature of the original C2f module, and achieving real-time inference in ultrasound scenarios where computational resources are scarce.

[0160] In one embodiment of the present invention, the cross-modal attention module includes three cross-modal attention units to perform cross-modal fusion on shallow feature map groups, mid-level feature map groups, and deep feature map groups, respectively.

[0161] When performing cross-modal fusion, the corresponding B-mode feature map is used as the master mode, and the corresponding blood flow mode feature map and elasticity mode feature map are guided by the cross-modal attention mechanism based on the Transformer structure.

[0162] When the cross-modal attention mechanism based on the Transformer structure guides the corresponding blood flow modal feature map and elastic modal feature map, the corresponding B modal feature map is used as the query source, and the corresponding blood flow modal feature map and elastic modal feature map are used as the key source and value source, respectively.

[0163] Figure 3The image illustrates one embodiment of the cross-modal attention module, which consists of three cross-modal attention units: Cross-modal Attention1, Cross-modal Attention2, and Cross-modal Attention3. Cross-modal Attention1 is used to perform cross-modal fusion on shallow feature map groups, Cross-modal Attention2 is used to perform cross-modal fusion on mid-level feature map groups, and Cross-modal Attention3 is used to perform cross-modal fusion on deep feature map groups. In practice, each cross-modal attention unit performs cross-modal fusion in the same way. For example, for the cross-modal attention unit Cross-modalAttention1, when performing cross-modal fusion on shallow feature maps, the shallow B-modal feature map is taken as the main modality. The cross-modal attention mechanism based on the Transformer structure guides the corresponding shallow blood flow modal feature map and shallow elastic modal feature map. Specifically, the shallow B-modal feature map is taken as the Query source, and the corresponding shallow blood flow modal feature map and shallow elastic modal feature map are taken as the Key source and Value source, respectively.

[0164] The following example uses the Cross-modal Attention1 unit to perform cross-modal fusion of shallow feature maps to illustrate the specific methods and processes of cross-modal fusion.

[0165] shallow B-mode feature map Superficial blood flow modal characteristics and shallow elastic modal feature map The data is loaded into the Cross-modal Attention Unit (Cross-modal Attention1). Cross-modal Attention1 transforms the two-dimensional feature map into a one-dimensional sequence, with each spatial location becoming a visual token. This sequence is then adapted to the Transformer input format via linear mapping and positional encoding to obtain the transformed feature map. The feature space of this feature map is... Where N represents the feature locations, i.e., the number of patches. For feature dimensions.

[0166] Since doctors typically prioritize B-mode images when observing ultrasound images, the B-mode is set as the dominant mode during cross-modal fusion, while the shallow B-mode feature map is used. As the primary modality feature map, the Cross-modal Attention mechanism based on the Transformer structure is used to guide the other two modalities, as shown in the following formula:

[0167]

[0168] in, It is a shallow cross-modal fusion feature map. Represents shallow B-mode feature maps As the source of the query, the superficial blood flow modality feature map Shallow elastic modal characteristic map They serve as the source of the Key and the source of the Value, respectively.

[0169] For superficial blood flow modal feature maps When performing Cross-Attention, we have:

[0170]

[0171] in, To process shallow B-mode feature maps using linear matrices A defined query vector is used to retrieve relevant information; To process shallow blood flow modal feature maps using linear matrices The key is used to calculate the feature sequence to be matched. To process shallow blood flow modal feature maps using linear matrices The value that follows. , , For a learnable linear projection matrix, the corresponding feature space is... , Shallow B-mode feature map From superficial blood flow modal characteristics The key position weights to focus on Shallow B-mode feature map superficial blood flow modal characteristics Enhanced features guided by the bottom.

[0172] Similarly, for shallow elastic modal characteristic maps When performing Cross-Attention, we have:

[0173]

[0174] in, To process shallow B-mode feature maps using linear matrices A defined query vector is used to retrieve relevant information; To process shallow elastic modal feature maps using linear matrices The key is used to calculate the feature sequence to be matched. To process shallow elastic modal feature maps using linear matrices The value that follows. , , For a learnable linear projection matrix, the corresponding feature space is... , Shallow B-mode feature map From the shallow elastic modal feature map The key position weights to focus on Shallow B-mode feature map Shallow elastic modal feature map Enhanced features guided by the bottom.

[0175] It should be understood that the aforementioned learnable linear projection matrix Learnable linear projection matrix Learnable linear projection matrix and learnable linear projection matrix Learnable linear projection matrix Learnable linear projection matrix All of these are determined after the model is trained and the lesion instance segmentation model is obtained.

[0176] As can be seen from the above description, the shallow B-mode feature map Superficial blood flow modal characteristics Shallow elastic modal characteristic map After guidance, shallow B-mode feature maps can be realized. Superficial blood flow modal characteristics and shallow elastic modal feature map The feature fusion yields:

[0177]

[0178] in, , These are weights that can be learned.

[0179] From the above explanation, we can conclude that: It is understandable that each cross-modal attention unit can adopt a commonly used form, specifically one that can satisfy the cross-modal fusion requirements described above.

[0180] In one embodiment of the present invention, the temporal feature fusion module includes three temporal feature fusion units, which are used to perform temporal fusion on all shallow cross-modal fusion feature maps, mid-level cross-modal fusion feature maps, and deep cross-modal fusion feature maps respectively.

[0181] During temporal fusion, a global instance query is determined on each cross-modal fusion feature map, and each global instance query is decomposed into frame-level features using a dynamic attention mechanism;

[0182] All corresponding frame-level features of the same type of cross-modal fusion feature map are concatenated to form a corresponding frame-by-frame concatenated query.

[0183] Each frame-by-frame concatenation query is weighted and aggregated to form the corresponding temporal fusion feature.

[0184] Figure 3 The diagram illustrates one embodiment of a temporal feature fusion module, comprising three temporal feature fusion units: Temporal Feature Fusion1, Temporal Feature Fusion2, and Temporal Feature Fusion3. Temporal Feature Fusion1 performs temporal fusion on all shallow cross-modal fusion feature maps, Temporal Feature Fusion2 performs temporal fusion on all mid-level cross-modal fusion feature maps, and Temporal Feature Fusion3 performs temporal fusion on all deep cross-modal fusion feature maps. In practice, each temporal feature fusion unit performs the same temporal fusion method, which can be the temporal fusion module in the SeqFormer model, and the temporal fusion method can be consistent with existing technologies.

[0185] Specifically, the temporal feature fusion unit includes a Transformer module, which adopts a structure of independent frame-by-frame and cross-frame aggregation. It locates the lesion target in each frame through cross-frame shared instance query. Then, the instance query in each frame is decomposed into target box query. These target box queries use an attention mechanism to integrate the box coordinate information in each frame to obtain the local feature of the instance corresponding to each frame. Then, they are fused into a video-level instance through aggregation to achieve following the trajectory of the instance target. Here, a frame specifically refers to a cross-modal fusion feature map.

[0186] The following example illustrates the temporal fusion method by performing temporal fusion on all shallow cross-modal fusion feature maps. Specifically,

[0187] Determine the learnable global instance queries on the shallow cross-modal fusion feature map of each frame, such as... ,in, For global instance querying, the meaning is consistent with existing technologies, such as learning a learnable embedding vector for each video-level instance to retrieve corresponding features across all frames. This refers to the number of queries at the video level; the number of instance queries used here is [number of queries per video]. The value is 300, where D represents the feature dimension of the global instance query.

[0188] If each global instance query is decomposed into frame-level features based on temporal sequence, then:

[0189]

[0190] in, Let P represent the shallow cross-modal fusion feature map of the input frame t, where P represents the P tokens flattened out, with a flattening dimension of D1. Indicates the t-th frame Frame-level feature bounding boxes located and extracted on the layer This indicates a dynamic attention module that calculates only local sampling points, significantly reducing complexity. Moreover, the offset can be learned, enabling dynamic tracking of the target. This is suitable for situations where the location of the lesion area changes during ultrasound scanning.

[0191] In practice, the flattening dimension D1 and the feature dimension D of the global query instance should have the same value, such as both being 256. As explained above, when the multimodal target time series data includes multiple sets of multimodal data to be inspected, the shallow cross-modal fusion feature of frame t can be determined based on the time series of the multimodal data to be inspected. However, when the multimodal target time series data contains only one set of multimodal data to be inspected, the shallow cross-modal fusion feature of frame t is the current shallow cross-modal fusion feature.

[0192] To improve efficiency, the video frame-level dynamic attention mechanism is parallelized, which means that global instance queries on all shallow cross-modal fusion feature maps are decomposed in parallel using temporal decomposition. When determining global instance queries, each global instance query is assigned a number. After parallelization, frame-level feature bounding boxes with the same number are directly concatenated to form corresponding frame-by-frame concatenated bounding box queries.

[0193] After obtaining the frame-by-frame concatenation query, a learnable weighted aggregation is performed on the frame-by-frame concatenation query, and the global instance query of the current frame is updated, resulting in:

[0194]

[0195] in, For the updated global instance query, For the t-th frame Frame-by-frame concatenation of target bounding boxes at the layer level. For the first Global instance query of the layer To perform frame-by-frame stitching of target bounding boxes for querying A feedforward network mapped to scalar weights is used to learn temporal weights, thus achieving temporal feature fusion. This represents the number of groups of multimodal data to be inspected within the current multimodal target time series data.

[0196] It should be noted that the structure of the scalar weighted feedforward network is related to the temporal feature fusion unit. The global instance query of the previous layer refers to the instance query before it is fed into the feedforward network. That is, when updating the global instance query, a residual connection is used. Generally, the feedforward network can be a two-layer multilayer perceptron (MLP), but other structural forms are also possible. After updating the global instance query of the frame in the above manner, the video-level representation of the previous frame is preserved, spatial positioning accuracy is enhanced, and robust temporal feature fusion is achieved.

[0197] In practical implementation, when shallow cross-modal fusion feature maps exist, the frame order of the shallow cross-modal fusion feature maps is determined according to the time sequence in which they are formed. For example, as mentioned above, if there are 5 sets of multimodal data to be inspected within the multimodal target time series data, then according to the corresponding time sequence, shallow cross-modal fusion feature maps corresponding to frame sequences 1 to 5 can be obtained. Frame sequence 1 corresponds to time sequence 1; therefore, the time sequence of frame sequence 5 is later than that of frame sequence 1. During time series fusion, a global instance query is defined on the shallow cross-modal fusion feature maps corresponding to frame sequences 1 to 5, and parallel processing and updating are performed as described above to obtain the corresponding shallow time series fusion features. It can be understood that when there is only one set of multimodal data to be inspected within the multimodal target time series data, updating the global instance query will form the corresponding shallow time series fusion features.

[0198] The neck network enables multi-scale fusion. To integrate with the temporal feature fusion module of this invention, the neck network includes a first upsampling unit connected to the Temporal Feature Fusion 3 unit and a fourth neck stitcher. The output of the first upsampling unit is connected to one input of the first neck stitcher, and the other input of the first neck stitcher is connected to the Temporal Feature Fusion 2 unit.

[0199] The output of the first stitcher in the neck is connected to the first convolutional-attention fusion module in the neck. The output of the second convolutional-attention fusion module in the neck is connected to the second upsampling unit and one input of the third stitcher in the neck. The second upsampling unit is connected to one input of the second stitcher in the neck. The other input of the second stitcher in the neck is connected to the temporal feature fusion unit (Temporal Feature Fusion2). The output of the second stitcher in the neck is connected to the first convolutional-attention fusion module in the neck. The output of the first convolutional-attention fusion module in the neck is connected to the segmentation output head and the neck convolutional layer.

[0200] The output of the neck convolutional layer is connected to the other input of the neck third stitcher. The output of the neck third stitcher is connected to the input of the neck third convolution-attention fusion module. The output of the neck third convolution-attention fusion module is connected to the segmentation output head and the neck spatial-channel decoupling downsampling module. The output of the neck spatial-channel decoupling downsampling module is connected to the neck fourth stitcher adapter. The neck fourth stitcher is connected to the segmentation output head adapter via the neck C2fCIB module.

[0201] Figure 4 The diagram illustrates an embodiment of the neck network. In the diagram, Upsample1 is the first upsampling unit, Concat1 is the first neck stitcher, C2f-ConvForm11 is the second neck convolution-attention fusion module, Upsample2 is the second upsampling unit, Concat2 is the second neck stitcher, C2f-ConvForm10 is the first neck convolution-attention fusion module, Conv7 is the neck convolutional layer, Concat3 is the third neck stitcher, C2f-ConvForm11 is the third neck convolution-attention fusion module, SCDown7 is the neck spatial-channel decoupling downsampling module, Concat4 is the fourth neck stitcher, and C2fCIB4 is the neck C2fCIB module.

[0202] Specifically, the specific structural forms of C2f-ConvForm10 to C2f-ConvForm12 can be found in the corresponding description of C2f-ConvForm1 above, and will not be repeated here. Furthermore, Figure 4The figure also illustrates an embodiment of the segmentation output head, which includes a prediction output head Head1, a prediction output head Head2, and a prediction output head Head3. The prediction output head Head1 is connected to the output end of C2f-ConvFormer10, the prediction output head Head2 is connected to the output end of C2f-ConvFormer12, and the prediction output head Head3 is connected to the output end of the neck C2fCIB module. Different scale prediction outputs can be achieved through the prediction output heads Head1, Head2, and Head3. Subsequently, the prediction outputs at different scales are post-processed to obtain the required lesion instance state information. The post-processing method can be consistent with the prior art and will not be described in detail here.

[0203] As explained above, when lesions are identified in the multimodal target time series data, the lesion instance status information should at least include the lesion category. Similar to existing technologies, the lesion category should be labeled on the multimodal target time series data. In specific implementation, when multiple multimodal data to be examined exist in the multimodal target time series data, the latest multimodal data to be examined is taken as the current data, and the B-mode image to be examined within the current data is taken as the master modality image. After determining the corresponding master modality image, the lesion category is labeled on the master modality image as the output of the lesion instance status information corresponding to the current multimodal target time series data.

[0204] Regarding the lesion instance segmentation model described above, the construction method and process are illustrated below. In one feasible embodiment, it includes:

[0205] Construct a basic model for lesion instance segmentation and build an instance segmentation training dataset for model training;

[0206] Configure instance segmentation training conditions, and train the basic model for lesion instance segmentation using the instance segmentation training dataset under the instance segmentation training conditions until the target training state is reached. The model that has reached the target training state and is configured as the aforementioned lesion instance segmentation model.

[0207] It should be understood that the basic model for lesion instance segmentation can use the same network structure as the lesion instance segmentation model described above. For details on the basic model for lesion instance segmentation, please refer to [link / reference needed]. Figures 2-5The corresponding explanations mentioned above will not be repeated here. The constructed instance segmentation training dataset should include several multimodal training time-series data. Each set of multimodal training time-series data should include multiple sets of multimodal training time-series samples. The multimodal nature of this data can be referred to in the descriptions of the multimodal target time-series data and the multimodal test data mentioned above. That is, each multimodal training time-series sample should include training B-mode images, training blood flow mode images, and training elasticity mode images. The generation method of each set of multimodal training time-series samples can be referred to in the descriptions of the multimodal test data mentioned above, which can be determined by methods such as sliding windows. Unlike the inference work mentioned above, each set of multimodal training time-series data should include multiple sets of multimodal training time-series samples, such as 5 sets of multimodal training time-series samples per set. The number of multimodal training time-series samples can be selected as needed, and will not be repeated here.

[0208] When preparing the instance segmentation training dataset, existing ultrasound images of organs from medical ultrasound scans can be labeled and generated. The organs involved in the instance segmentation training dataset form a training organ group, which may include organs such as the breast, thyroid, liver, carotid artery, and gallbladder. It should be understood that when lesions are identified in the multimodal target time-series data after lesion segmentation and identification processing, the lesion category should be one of the organ categories within the training organ group. In specific implementation, the ultrasound images used should include scan data containing lesions and normal scan data without obvious abnormalities. Both scan data containing lesions and normal scan data without obvious abnormalities should include B-mode images, blood flow modal images, and elastic modal images. The methods and processes for obtaining B-mode images, blood flow modal images, and elastic modal images can be referred to the corresponding descriptions above.

[0209] In practice, when the frame rates of the B-modal images, blood flow modal images, and elasticity modal images are inconsistent, the synchronization of different modal data on the time axis is ensured. This involves unifying the frame rate and repairing missing frames. Specifically, when the frame rates of different modalities are inconsistent, linear interpolation is used to automatically fill in frames for the modality with the lower frame rate. If some modalities have missing frames, adjacent frame interpolation is used to repair the missing frames to ensure temporal integrity. This ensures that the training B-modal images, training blood flow modal images, and training elasticity modal images in each group of multimodal training time-series samples have consistent temporal sequence, meaning they can all represent the corresponding scanning state under the same time sequence. In addition, corresponding image preprocessing should also be performed when training B-modal images, training blood flow modal images, and training elasticity modal images. The image preprocessing methods and processes used can be referred to the above descriptions, and will not be repeated here.

[0210] It should be understood that when lesions are present, commonly used annotation methods should be used to annotate sample features on the training B-mode image, training blood flow modality image, and training elasticity modality image within the corresponding multimodal training time-series samples. Sample features include the type of lesion and the segmentation mask of the lesion contour. When no lesions are present, no sample feature annotation is performed within the corresponding multimodal training time-series samples. Annotation can be performed on the B-mode image, which serves as the primary modality image within the current multimodal training time-series samples. The meaning of the primary modality image can be found in other relevant explanations.

[0211] Generally, instance segmentation training conditions should include the optimizer used for model training, the loss function for segmentation model training, and some necessary model training parameters. Necessary model training parameters may include the number of iterations in training, etc. Necessary model training parameters can be selected as needed to meet the requirements of model training. Examples will not be given here.

[0212] The training loss function for segmentation models can include classification loss. Target bounding box regression loss Mask loss and timing loss Then we have:

[0213]

[0214] in, To train the loss function for the segmentation model, These are the weighting coefficients for the classification loss. These are the weighting coefficients for the target bounding box regression loss. These are the weighting coefficients for the mask loss. The weighting coefficients for the time series loss.

[0215] It should be noted that the weighting coefficients mentioned above can be dynamic and adjusted based on the training phase. For example, if the mask prediction is more difficult in the early stages of training, the weighting coefficients of the mask loss can be adjusted. The setting can be relatively large, such as 0.8. As training progresses and the prediction of instance classes becomes more accurate, i.e., after the classification loss stabilizes, the weight coefficient of the classification loss can be reduced. The weights of the remaining loss functions are increased. Dynamic weights enable the model to adapt to datasets from different ultrasound scan sites, and based on the difficulty, ultimately achieve joint optimization of temporal instance segmentation. The dynamic selection method is chosen as needed, and will not be elaborated here.

[0216] In practice, during model training, multimodal training time-series samples are loaded into the lesion instance segmentation basic model one by one or in batches. The lesion instance segmentation basic model then outputs a lesion training prediction information. Similar to the lesion instance state output by the working inference, when a lesion exists in the multimodal training time-series sample, the lesion training prediction information should include the lesion category, the detection box of the lesion region, and the segmentation mask of the lesion region. Furthermore, the lesion training prediction information should also be annotated on the corresponding training B-mode image in the latest multimodal training time-series sample within the multimodal training time-series data. In other words, the training B-mode image of the latest multimodal training time-series sample is still used as the main modality image during training.

[0217] Calculate classification loss When this is the case, a binary cross-entropy loss with sigmoid can be used; therefore, the classification loss is calculated. The commonly used binary cross-entropy loss formula can be used. When calculating the binary cross-entropy loss, the predicted lesion category for each multimodal training time-series sample and the lesion category labeled on the corresponding main modality image are mainly used. Then, by substituting the corresponding category information into the formula, the corresponding classification loss can be calculated. .

[0218] Calculate the target bounding box regression loss function When calculating the full crossover ratio loss, the full crossover ratio loss can be used. The calculation expression of the full crossover ratio loss can adopt the existing commonly used methods. When calculating the full crossover ratio loss, the crossover ratio of the detection box of the lesion predicted by each multimodal training time sequence sample and the detection box of the corresponding labeled lesion on the corresponding main modality image is mainly used. The specific calculation method and process will not be elaborated here.

[0219] Calculate the mask loss function When using a pixel-level binary classification loss function loss function with mask overlap rate The weighted average is calculated using the following formula:

[0220]

[0221] in, pixel-level binary classification loss function The weighting coefficients, For mask overlap rate loss function The weighting coefficients.

[0222] In practice, the pixel-level binary classification loss function is calculated. The primary method utilizes the foreground / background classification (binary classification of foreground or background) of pixels within the lesion contour region predicted from multimodal training time-series samples, along with the foreground / background classification of pixels within the lesion contour region labeled on the main modality image, to calculate the corresponding pixel-level binary classification loss. The mask overlap rate loss function primarily uses the lesion instance mask predicted from multimodal training time-series samples and the mask of the lesion labeled on the main modality to calculate the degree of overlap between the two masks, thus determining the degree of non-overlap and obtaining the mask overlap rate loss.

[0223] Calculate timing loss At that time, the loss is calculated based on the difference in output results, as shown in the following formula:

[0224]

[0225] in, The number of multimodal training time-series data points within the training dataset is used to segment instances. This represents the total number of frames in the instance segmentation training dataset that can form a temporal sequence. The lesion type predicted by the basic model for lesion instance segmentation on the m-th multimodal training time-series sample. The lesion type predicted by the basic model for lesion instance segmentation on the (m+1)th multimodal training time series sample. The difference metric function is represented here. The L2 distance function is chosen to ensure that the mask of the same target changes smoothly between frames and to constrain the continuity of the target's position and shape. By constraining feature consistency, the temporal stability of the final output is indirectly guaranteed, and the model is prevented from learning noise features that are irrelevant between frames.

[0226] It should be noted that the total number of frames in the time series is related to the number of multimodal training time series data and multimodal training time series samples. As explained above, using the sliding window operation, the images corresponding to frames 1 to 5 form multimodal time series data, and the multimodal time series data formed by the images corresponding to frames 2 to 6 can constitute the number of frames in a time series. Other cases can be referred to here.

[0227] It should be understood that during model training, each multimodal training time series should include multiple multimodal training time series samples. However, during inference, if each multimodal target time series contains only one multimodal test data, it will not affect the inference work of the basic model for lesion instance segmentation.

[0228] Similar to existing models, the basic model for lesion instance segmentation is iteratively trained using the instance segmentation training dataset. When the number of iterations reaches the target number of iterations, the iterative training is terminated, thus reaching the target training state.

[0229] In one embodiment of the present invention, when there are two temporally correlated multimodal target time-series data, and lesion segmentation and identification processing determines that lesions exist in each multimodal target time-series data, then lesion tracking is performed on the lesions identified corresponding to the two temporally correlated multimodal target data.

[0230] Lesion tracking includes:

[0231] For each multimodal target time series data, obtain the lesion detection box for each lesion after lesion segmentation and identification processing;

[0232] The lesion detection box of each lesion in the previous time series multimodal target time series data is used as the basic tracking box, and the image region corresponding to the basic tracking box is configured as the basic tracking region. At the same time, the lesion detection box of each lesion in the next time series multimodal target time series data is used as the detection box to be matched, and the image region corresponding to each detection box to be matched is configured as the region to be matched.

[0233] Motion prediction is performed on the basic tracking boxes to generate basic tracking prediction boxes. Then, Hungarian matching detection is performed between the basic tracking prediction boxes and each target detection box.

[0234] When the basic tracking prediction box and the target detection box satisfy the Hungarian matching, the basic tracking prediction box is configured as the tracking target box, and the corresponding target detection box is configured as the tracking matching box.

[0235] When the tracking target box does not meet the tracking matching box that satisfies the Hungarian matching, the image region corresponding to the tracking target box is subjected to foreground target supplementation segmentation on the main modality map of the current frame group of multimodal data to be inspected in the subsequent time series multimodal data, so as to determine the lesion information of the image region corresponding to the tracking target box after foreground target supplementation segmentation.

[0236] During ultrasound scanning, multiple lesions may exist simultaneously in organs such as the thyroid, breast, or liver. As the scanning position and angle of the probe change, the lesions may shift, deform, or be temporarily obscured. In such cases, the lesion instance segmentation model may lose the original index of the target or lose the target completely. To minimize this situation, this invention performs lesion tracking to distinguish multiple lesions and continuously track the changes of each lesion.

[0237] It should be noted that when tracking lesions, the focus should be on two adjacent multimodal target time-series data generated during the same ultrasound scan. Therefore, the temporal correlation mentioned above specifically refers to the temporal characteristics of two multimodal target time-series data. For example, in one ultrasound scan, 10 sets of multimodal data to be examined can be generated. When the sliding window length is 5, the first multimodal target time-series data can be generated based on the multimodal data to be examined with time series 1 to 5, and the second multimodal target time-series data can be generated based on the multimodal data to be examined with time series 2 to 6. When tracking lesions, the focus should be on the two multimodal target time-series data in this example. In this case, the time series of the first multimodal target time-series data corresponds to the previous time series, and the time series of the second multimodal target time-series data corresponds to the next time series.

[0238] It should be understood that during lesion tracking, the state information of the lesion instance corresponding to each multimodal target time series data should be obtained. For example, it should be determined that at least one lesion exists within the multimodal target time series data of the previous time series; otherwise, lesion tracking is unnecessary. As explained above, for each lesion instance state information, after determining the lesion category, the lesion detection box can also be determined. The lesion detection box is the area and size that marks the corresponding outline of the lesion. The location of each lesion can be determined through the lesion detection box, and the configuration of the lesion detection box can be consistent with existing technologies.

[0239] In lesion tracking, the lesions corresponding to the previous time series are typically tracked in conjunction with the lesions corresponding to the next time series. Therefore, the lesion detection bounding box of each lesion in the previous time series multimodal target time series data is used as the basic tracking bounding box, and the image region corresponding to the basic tracking bounding box is configured as the basic tracking region. Simultaneously, the lesion detection bounding box of each lesion determined in the next time series multimodal target time series data is used as the detection bounding box to be matched, and the image region corresponding to each detection bounding box to be matched is configured as the region to be matched. In the lesion tracking of this invention, each basic tracking bounding box and the detection bounding box to be matched are first subjected to Hungarian matching. If a Hungarian matching is satisfied, then tracking matching is performed. If a Hungarian matching is satisfied, the basic tracking bounding box is configured as the tracking target bounding box, and the basic tracking region is configured as the tracking target region.

[0240] In one embodiment of the present invention, when performing Hungarian matching detection between the basic tracking region and the region to be matched, the method includes:

[0241] The basic tracking bounding boxes are preprocessed using Hungarian matching to generate basic tracking prediction bounding boxes.

[0242] The Hungarian matching preprocessing includes appearance feature extraction, smoothing, and / or Kalman filtering motion prediction.

[0243] Based on the appearance and area of ​​the basic tracking prediction box and the corresponding detection box to be matched, the cost matrix is ​​calculated.

[0244] Based on the calculated cost matrix, the Hungarian matching algorithm is used to solve the Hungarian matching problem to determine the Hungarian matching status between the basic tracking region and the corresponding preprocessing region to be matched.

[0245] Specifically, when performing Hungarian matching detection, the BoT-SORT-ReID algorithm can be used. In addition to using a Kalman filter for motion prediction, BoT-SORT-ReID also enables Re-ID to maintain the consistency of the target index using appearance features. ReID is a network model that extracts the appearance feature vector of each detected target box. That is, when performing Hungarian matching through the BoT-SORT-ReID algorithm, the basic tracking box can be preprocessed with Hungarian matching and the corresponding basic tracking prediction box can be generated.

[0246] It should be understood that in multi-target tracking, relying solely on IOU calculation and Kalman motion prediction cannot reliably distinguish between targets with similar appearances, intersecting trajectories, or occlusions. Therefore, vectors are extracted for each lesion detection box and historical trajectory, and similarity is calculated to obtain a cost matrix. When calculating the cost matrix, based on the appearance and area of ​​the basic tracking prediction box and the corresponding detection box to be matched, the intersection-union ratio (IoU) distance and cosine distance between the basic tracking prediction box and the detection box to be matched are calculated. The calculated IoU distance and cosine distance can be calculated using techniques commonly used in this field. The specific method and process for calculating the cost matrix are consistent with existing technologies and will not be elaborated here.

[0247] After calculating the cost matrix, the Hungarian matching algorithm is used to solve for the Hungarian matches. The Hungarian matching state can be either satisfied or not satisfied. The method and process of solving for the Hungarian matches based on the cost matrix are consistent with existing techniques and will not be elaborated here. When a Hungarian match is satisfied, it indicates that the basic tracking region and the region to be matched belong to the same lesion; otherwise, the basic tracking region and the region to be matched belong to different lesions. When the basic tracking region and the region to be matched belong to the same lesion, it indicates that correct tracking has been performed. It is understandable that when a basic tracking region does not satisfy a Hungarian match with any of the regions to be matched, it indicates that there is no region belonging to the same lesion as the basic tracking region in the subsequent time series.

[0248] As can be seen from the above description, after satisfying the Hungarian matching, a follow-up matching judgment should also be performed. In one embodiment of the present invention, the follow-up matching judgment includes:

[0249] Calculate the Intersection over Union (IOU) between the target bounding box and the tracking matching bounding boxes that satisfy the Hungarian matching.

[0250] When the calculated Intersection over Union (IOU) meets the tracking matching threshold, the tracking target box and the tracking matching box that meets the Hungarian matching are considered to be matched. Otherwise, the tracking target box and the tracking matching box that meets the Hungarian matching are not considered to be matched.

[0251] When performing tracking and matching judgment, this invention mainly utilizes the Intersection over Union (IOU) between the tracking target bounding box and the tracking matching bounding box that satisfies the Hungarian matching. Specifically, after calculating the corresponding IOU using common techniques in this technical field, the calculated IOU is compared with a tracking matching threshold. If the calculated IOU satisfies the tracking matching threshold, then the tracking target bounding box and the tracking matching bounding box that satisfies the Hungarian matching are considered to be tracking matched; otherwise, the tracking target bounding box and the tracking matching bounding box that satisfies the Hungarian matching are not considered to be tracking matched. The IOU satisfying the tracking matching threshold specifically means that the calculated IOU is the same as the tracking matching threshold, or the difference between the two is within an allowable numerical range, which can be selected as needed.

[0252] The tracking matching threshold can be set as needed, based on the criteria for determining tracking matching. Custom settings can be made for different organs; a typical threshold is 0.7. It is understood that when the basic tracking region and the region to be matched simultaneously satisfy both Hungarian matching and tracking matching, it indicates that the basic tracking region and the region to be matched belong to the same region. However, the contours of the corresponding lesions in the previous and subsequent time series may differ. When they differ, it means that a lesion may be missed.

[0253] To avoid missed detections, in one embodiment of the present invention, foreground target supplementation segmentation is performed on the main modality image of the current frame group of multimodal data to be detected within the subsequent time-series multimodal data. This foreground target supplementation segmentation is used to determine the lesion information in the image region corresponding to the tracking target box. When performing foreground target supplementation segmentation, commonly used existing foreground target segmentation models can be used, such as the PPLiteSeg-STDC2 model. After foreground target supplementation segmentation, the results should be output on the main modality image. The results of foreground target supplementation segmentation may include the lesion category and the corresponding contour of the lesion.

[0254] When there are multiple temporally correlated multimodal target time series data, the above method is used to track lesions for two adjacent multimodal target time series data, that is, continuous lesion tracking can be achieved during ultrasound scanning.

[0255] In one embodiment of the present invention, when it is determined that there is a lesion in the multimodal target time series data after lesion segmentation and identification processing, the lesion instance status information is marked on the main modality map of the current frame group of multimodal data to be inspected in the current multimodal target time series data;

[0256] Based on the main modality map of the multimodal data to be detected in the frame group and the lesion detection box marked on the main modality map for each lesion, lesion feature description processing is performed on each lesion to determine the lesion region features of each lesion. The lesion region features include echo type, shape and / or edge features.

[0257] As explained above, the state of a lesion instance can only determine its category. Therefore, to assist doctors or researchers in diagnostic analysis, image processing algorithms are needed to quantify the detailed features of the lesions. In practice, the attributes required to describe various lesions in different organ locations differ, typically including common items such as echo type, shape, and edge features, while also potentially including unique descriptive items specific to certain organs. Therefore, specific analysis is required for various lesions in different organs. The following section uses breast lesions as an example to introduce descriptive algorithms related to lesion region features.

[0258] Specifically, descriptions of breast lesions typically include echo type, border type, edge features, calcification type, growth direction, posterior acoustic features, and shape type, among which...

[0259] When the lesion area is characterized by echoes, the echo type can be determined using techniques commonly used in this field. One feasible approach is as follows: First, perform grayscale histogram analysis on the region of interest of the lesion in the main modality image to extract grayscale distribution features. Second, calculate the average grayscale ratio between the lesion area and the surrounding normal tissue. Third, also statistically analyze the proportion of bright areas and the proportion of tissueless areas within the lesion. Specifically, when calculating the proportion features, the proportion of pixels in bright areas within the lesion to the total number of pixels in the entire lesion is calculated. In particular, it is determined whether the grayscale value of the current pixel matches a preset grayscale threshold. If it matches the preset grayscale threshold, the current pixel is considered bright; otherwise, it should be considered low brightness.

[0260] Based on the aforementioned grayscale distribution characteristics, average grayscale ratio, and proportion characteristics, the echo type can be determined. The echo type can be anechoic, hypoechoic, hyperechoic, isoechoic, or very hypoechoic. It should be noted that the method for determining the echo type can be consistent with existing technologies. For example, the grayscale distribution characteristics, average grayscale ratio, and proportion characteristics can be fused, and the fused result can be compared with an echo judgment threshold. The fusion method and the echo judgment threshold can be selected as needed; further examples are not provided here.

[0261] When the lesion region is characterized as a boundary type, a feasible method for determination is to calculate the boundary gray-level gradient of the lesion region to obtain the average gray-level separation. Then, based on the calculated average gray-level separation, the clarity of the boundary within the region is judged. Generally, the boundary type is classified as clear or blurred. In specific implementation, the methods for calculating the boundary gray-level gradient and average gray-level separation can be consistent with existing technologies. Furthermore, the method for judging the clarity of the boundary within the region based on the calculated average gray-level separation can also be consistent with existing methods, and will not be elaborated upon here.

[0262] When the lesion region features are edge features, a feasible method for determination is to use methods such as lesion expansion detection, convexity analysis, and polygon approximation to determine whether the edge is smooth. Among them, lesion expansion detection is used to determine whether the lesion outline breaks through the organ capsule or conflicts with the organ boundary. This step has the highest priority. Once the above situation is detected, the result of non-smoothness is directly obtained. Convexity analysis includes the calculation of the contour indentation depth, maximum indentation, and number of indentations. Polygon approximation uses the Douglas-Peucker algorithm to simplify the contour.

[0263] When the lesion area is characterized by calcification, a feasible way to determine it is to fill in the outline of the calcified area, count the number of pixels in the calcified area, detect whether there is acoustic shadowing, and calculate the ratio of the calcified area to the total lesion area to determine the degree of calcification. The degree of calcification can be divided into no calcification, coarse calcification and microcalcification.

[0264] In practice, the calcified area can be determined by binarizing the lesion area. Once the calcified area is determined, its outline can be filled. Then, commonly used techniques in this field can be employed to determine if acoustic shadowing exists. Further, after determining the calcified area, the judgment logic includes judging the area ratio. If it is greater than the area threshold Thresh1, it is judged as coarse calcification; if there is acoustic shadowing, it is directly judged as coarse calcification; if the number of calcifications is greater than the number threshold N and the area ratio is less than Thresh2, it is judged as microcalcification. The thresholds Thresh1, Thresh2, and N are adjusted according to different organ types. The specific adjustment method can be consistent with existing technologies and will not be elaborated here.

[0265] When the lesion region is characterized by its growth direction, a feasible method for determination is as follows: The principal component analysis (PCA) is used to calculate the main direction of the lesion as its major axis, and the direction perpendicular to the main direction is obtained as its minor axis. The inner diameter of the lesion is then calculated by combining this with the lesion contour. The method for calculating the inner diameter can be consistent with existing techniques. Specifically, the inner diameter includes both the major and minor axes. After obtaining the inner diameter, the angle between the major axis and the horizontal direction is calculated, and then the ratio of the major to minor axis of the inner diameter is calculated. The ratio of the major to minor axis and the angle are used to determine whether the lesion's growth direction is parallel to the horizontal direction.

[0266] When the lesion region features posterior acoustic features, a feasible method for determination is as follows: First, define the posterior direction of the lesion. Since the posterior direction varies slightly for different probes, after obtaining the region of interest behind the lesion, perform acoustic feature analysis, detect the dark area with lower grayscale and the bright area with higher grayscale behind the lesion, and determine whether the acoustic features behind the current lesion are attenuated, enhanced, or unchanged based on the positional relationship and grayscale difference between the dark and bright areas.

[0267] When the lesion region features a shape type, a feasible method for determination is as follows: first, fit an ellipse to the lesion outline based on the minimum bounding rectangle and calculate the ratio of the major axis to the minor axis; then calculate the aspect ratio of the lesion boundary box to help determine the width and height directions; and finally, combine the concave features of the lesion outline to determine whether the lesion shape is circular, elliptical, or irregular.

[0268] In one embodiment of the present invention, the generated lesion instance status information is loaded into the constructed diagnostic report generation model, so as to generate a corresponding diagnostic report using the diagnostic report generation model, wherein,

[0269] When building a diagnostic report generation model, the following are included:

[0270] A large language basic model is provided, and the large language basic model is fine-tuned using the constructed large language model training dataset. During fine-tuning, the QLoRA method is used.

[0271] After training the basic large language model to the target state using the large language model training dataset, a diagnostic report generation model is formed based on the trained basic large language model.

[0272] When generating a diagnostic report, the diagnostic report generation model processes the input lesion instance status information and combines it with retrieval enhancement to generate the corresponding diagnostic report.

[0273] After obtaining the lesion instance status information and lesion region features, it is necessary to comprehensively consider all results and eliminate logically flawed ones. Large Language Models (LLMs), trained on massive amounts of text data, absorb knowledge including medical and image processing expertise. This knowledge can be integrated and transferred, enabling the LLM to accurately understand the meaning of professional medical terminology and image feature descriptions within the lesion instance status information and lesion region features, and generate analysis results in a logically clear text format, thus achieving intelligent diagnostic report generation.

[0274] In practical implementation, the basic large language model can be Llama-3.1-8B-Instruct. When constructing the large language model training dataset, the data used can include Huatuo-26M and a custom dataset. The custom dataset contains structured and unstructured data. Specifically, structured data includes existing case reports and prescription information, while unstructured data includes medical history records, image text reports, and doctor's notes. When the data is medical data, the following preprocessing operations are required before annotation. These preprocessing operations mainly include desensitization, medical terminology standardization, and label consistency checks. Desensitization requires removing sensitive information such as patient names, hospitals, dates, and medical record numbers. Medical terminology standardization mainly maps non-standardized medical terms to general standardized text. Label consistency checks mainly require manual label verification to avoid ambiguity.

[0275] After the custom dataset is annotated, a custom word segmenter specifically for medical terminology is also required. Because general-purpose word segmenters cannot understand medical terms such as "carotid atherosclerotic plaque" or fully recognize the overall semantics in diagnosis, directly using general-purpose word segmenters for medical terms may result in failure.

[0276] To address the issue of semantic invalidity, this invention uses SentencePiece as a word segmentation tool to merge high-frequency medical professional corpora into new sub-words and store them in a new dictionary. This facilitates the understanding of complete term meanings through sub-word combinations by the large language model. In other words, the speech model data samples in the large language model training dataset are segmented and merged using a word segmenter to form language model training samples. The construction of language model training samples and the state of each language model training sample can be consistent with the existing ones, which will not be elaborated here.

[0277] When training a large language base model, fine-tuning using the QLoRA method involves freezing the pre-trained weights of the large language base model and inserting learnable low-rank matrices only into the key layers of the large language base model. and low-rank matrices Using the query matrix and value matrix configured as the multi-head attention mechanism within the basic model of a large language, we have:

[0278]

[0279] in, For the pre-trained weights of the basic language model, Fine-tuning the weights for the basic model of a large language.

[0280] When a learnable low-rank matrix is ​​inserted into the multi-head attention mechanism within the basic large language model... and low-rank matrices Subsequently, it is able to capture contextual dependencies and semantic associations, and only the low-rank matrix is ​​updated during training. and low-rank matrices The number of parameters required for training is only 0.1% of that of the pre-trained model, significantly reducing computational resource requirements while avoiding impacting the general knowledge already learned by the larger model. Understandably, this involves inserting a learnable low-rank matrix into the multi-head attention mechanism within the basic language model. and low-rank matrices The method can be consistent with existing technologies, and will not be elaborated here.

[0281] In practice, during fine-tuning, the loss function can include cross-entropy loss. Knowledge conflict penalty loss Then we have:

[0282]

[0283] in, To balance cross-entropy loss Penalty for knowledge conflict Hyperparameters.

[0284] It should be noted that when training a large language basic model, given the context, it predicts the next word (token). The prediction is a probability distribution, and the predicted value is the predicted probability of the next token. The true value is the training prediction, i.e., the actual next token. Therefore, the commonly used cross-entropy loss can be used. The cross-entropy loss within the loss function is calculated using this method. The specific calculation method and process will not be elaborated here.

[0285] Calculate the penalty loss for knowledge conflict When calculating the loss primarily based on the conflict between the output of the current large language basic model and known knowledge, we have:

[0286]

[0287] in, This is the set of conflicts between the word segmentation sequence output by the basic language model and the known knowledge tokens. Represents the generation of the basic model of a large language up to the th When segmenting words, predict the probability of a conflict.

[0288] In the above set of token conflicts, a conflict is defined as the output of the basic language model that contradicts the content of the medical knowledge base. For the first Each word, This indicates conditional input, such as prompts or context. For the first Each word Enter conditions The predicted probability is as follows.

[0289] During fine-tuning training, the optimizer used is AdamW; the learning rate is scheduled using cosine annealing; and early stopping is employed to avoid semantic shift. Once the training of the large language base model reaches the target state, the required diagnostic report generation model can be formed based on the large language base model. Reaching the target training state can be achieved by reaching a preset number of iterations; other target training states can also be set, which will not be elaborated upon here.

[0290] During reasoning, the diagnostic report generation model needs to be combined with retrieval-enhanced generation (RAG) for intelligent analysis. In specific implementation, this invention uses RAG to output the corresponding diagnostic report, where retrieval-enhanced generation mainly includes: knowledge base construction, medical semantic encoding, retrieval selection, retrieval generation concatenation, generator reasoning optimization, and post-processing verification.

[0291] The most crucial aspect is the construction of the knowledge base. This invention integrates clinical guidelines, the latest research papers in the field of ultrasound, medical textbooks, and newly added case samples to maximize the use of existing key data and improve the accuracy of the model's disease analysis.

[0292] Medical semantic encoding is used to enable the retrieval system to understand the semantics of medical-related content. Therefore, a medical-specific encoding method is required. The medical-specific model PubMedBGE used in this invention employs sentence vector normalization as an optimization strategy to stabilize cosine similarity retrieval.

[0293] The retrieval system is responsible for finding relevant documents from the knowledge base. The retrieval strategy is to select the N documents that are most relevant to the known prompt question. Here, N can be set to 5.

[0294] The retrieval generation and splicing process uses a structured template, which clearly distinguishes between the retrieved information and the user's question. This prevents the model from confusing factual context with the generation task when recognizing the input text, and can significantly reduce the proportion of non-factual content in the model.

[0295] The generator inference optimization involves adjusting many parameters. Among them, temperature T is used to control the randomness of the generated text, and is set to 0.3 here. Top-p kernel sampling controls the probability of selecting the next word in the model output, avoiding excessive concentration of the model output on a small number of words and preventing the model from being too random and unrelated. Since the current application scenario is medical auxiliary diagnosis, the requirements for the output results are more rigorous, so it is set to 0.95 here. Due to the requirement of refining the diagnostic results, this invention also limits the maximum tokens output length, which is set to the range of [256, 512] here.

[0296] Post-processing verification is an important guarantee for enhancing the security of model output. This invention uses logical consistency confirmation and setting confidence score thresholds to filter results. Logical consistency confirmation uses some established rules to avoid ambiguity in diagnostic results, such as the logical conflict of "hypoechoic" and "hyperechoic" coexisting in the same lesion. Setting confidence score thresholds evaluates each knowledge base reference, and if it is below the threshold, the conclusion of that part is not displayed.

[0297] After RAG processing, the model output serves as the diagnostic report. The report may include the following: the organs examined, the reason for examination, the examination method, the comprehensive disease results of all ultrasound images, the corresponding descriptive and measurement items for the disease, and the existence of internationally used grading standards for some diseases. The type of organ examined can be determined based on the status information of the lesion instance. The reason for examination and the examination method can be generated by combining existing medical knowledge, such as the BI-RADS grading of breast ultrasound and the TI-RADS grading of thyroid ultrasound, comparison of follow-up information, and doctor's recommendations. The content of the diagnostic report can be selected as needed, and examples will not be given here.

Claims

1. A lesion identification method based on multi-modal ultrasound timing data, characterized in that, The lesion recognition method comprises: acquiring multi-modal target time-series data generated by performing an ultrasound scan on an organ, the multi-modal target time-series data comprising at least one set of multi-modal to-be-inspected data, wherein, when the multi-modal target time-series data comprises multiple sets of multi-modal to-be-inspected data, the multiple sets of multi-modal to-be-inspected data are sequentially generated according to the time sequence of the ultrasound scan; each set of multi-modal to-be-inspected data comprises at least a to-be-inspected B-mode image, a to-be-inspected blood flow mode image, and a to-be-inspected elasticity mode image representing the scanning state at the same time sequence; loading the multi-modal target time-series data into a lesion instance segmentation model to perform lesion segmentation and recognition processing on the multi-modal target time-series data using the lesion instance segmentation model, and outputting lesion instance state information corresponding to the current multi-modal target time-series data, wherein, during the lesion segmentation and recognition processing, multi-level feature extraction processing and cross-modal fusion feature extraction processing are sequentially performed on each set of multi-modal to-be-inspected data to generate cross-modal fusion feature information corresponding to each set of multi-modal to-be-inspected data, and thereafter, time sequence feature fusion processing, multi-scale fusion processing, and prediction output processing are sequentially performed on the cross-modal fusion feature information of all sets of multi-modal to-be-inspected data information, and the lesion instance state information is generated after the prediction output processing; when it is determined that there is a lesion in the multi-modal target time-series data after the lesion segmentation and recognition processing, the lesion instance state information at least comprises the category of each lesion; the lesion instance segmentation model comprises a backbone network, a cross-modal attention module, a time sequence feature fusion module, a neck network, and a segmentation output head connected in sequence, wherein, during the lesion segmentation and recognition processing, the backbone network is used to perform multi-level feature extraction processing on each set of multi-modal to-be-inspected data to form at least a shallow feature map group, a middle feature map group, and a deep feature map group, wherein, the shallow feature map group comprises a shallow B-mode feature map, a shallow blood flow mode feature map, and a shallow elasticity mode feature map; the middle feature map group comprises a middle B-mode feature map, a middle blood flow mode feature map, and a middle elasticity mode feature map; the deep feature map group comprises a deep B-mode feature map, a deep blood flow mode feature map, and a deep elasticity mode feature map; the cross-modal attention module is used to perform cross-modal fusion feature extraction processing to perform cross-modal fusion on the shallow feature map group, the middle feature map group, and the deep feature map group, respectively, and generate a shallow cross-modal fusion feature map, a middle cross-modal fusion feature map, and a deep cross-modal fusion feature map, respectively, and form the cross-modal fusion feature information of the current set of multi-modal to-be-inspected data based on the shallow cross-modal fusion feature map, the middle cross-modal fusion feature map, and the deep cross-modal fusion feature map; the time sequence feature fusion module is used to perform time sequence feature fusion processing on the cross-modal fusion feature information corresponding to all multi-modal target time-series data, wherein, during the time sequence feature fusion processing, all shallow cross-modal fusion feature maps are time sequence fused to generate a shallow time sequence fusion feature, all middle cross-modal fusion feature maps are time sequence fused to generate a middle time sequence fusion feature, and all deep cross-modal fusion feature maps are time sequence fused to generate a deep time sequence fusion feature; The neck network is used to perform multi-scale fusion of shallow, mid-level and deep temporal fusion features. After multi-scale fusion, the feature is processed by the segmentation output head to generate the corresponding lesion instance status information.

2. The method of claim 1, wherein the method further comprises: When two temporally correlated multimodal target time series data exist, and lesion segmentation and identification processing determines that lesions exist within each multimodal target time series data, then lesion tracking is performed on the lesions identified in the two temporally correlated multimodal target data. Lesion tracking includes: For each multimodal target time series data, obtain the lesion detection box for each lesion after lesion segmentation and identification processing; The lesion detection box of each lesion in the previous time series multimodal target time series data is used as the basic tracking box, and the image region corresponding to the basic tracking box is configured as the basic tracking region. At the same time, the lesion detection box of each lesion in the next time series multimodal target time series data is used as the detection box to be matched, and the image region corresponding to each detection box to be matched is configured as the region to be matched. Motion prediction is performed on the basic tracking boxes to generate basic tracking prediction boxes. Then, Hungarian matching detection is performed between the basic tracking prediction boxes and each target detection box. When the basic tracking prediction box and the target detection box satisfy the Hungarian matching, the basic tracking prediction box is configured as the tracking target box, and the corresponding target detection box is configured as the tracking matching box. When the tracking target box does not meet the tracking matching box that satisfies the Hungarian matching, the image region corresponding to the tracking target box is subjected to foreground target supplementation segmentation on the main modality map of the current frame group of multimodal data to be inspected in the subsequent time series multimodal data, so as to determine the lesion information of the image region corresponding to the tracking target box after foreground target supplementation segmentation.

3. The method of claim 2, wherein the method further comprises: When performing Hungarian matching detection between the basic tracking region and the region to be matched, the following is included: The basic tracking bounding boxes are preprocessed using Hungarian matching to generate basic tracking prediction bounding boxes. The Hungarian matching preprocessing includes appearance feature extraction, smoothing, and / or Kalman filtering motion prediction. Based on the appearance and area of ​​the basic tracking prediction box and the corresponding detection box to be matched, the cost matrix is ​​calculated. Based on the calculated cost matrix, the Hungarian matching algorithm is used to solve the Hungarian matching problem to determine the Hungarian matching status between the basic tracking region and the corresponding preprocessing region to be matched. 4.The method of claim 2, wherein, When performing tracking and matching judgments, the following are included: Calculate the Intersection over Union (IOU) between the target bounding box and the tracking matching bounding boxes that satisfy the Hungarian matching. When the calculated Intersection over Union (IOU) meets the tracking matching threshold, the tracking target box and the tracking matching box that meets the Hungarian matching are considered to be matched. Otherwise, the tracking target box and the tracking matching box that meets the Hungarian matching are not considered to be matched.

5. The method of claim 1, wherein the method further comprises: When the presence of a lesion is determined in the multimodal target time series data through lesion segmentation and identification processing, the lesion instance status information is marked on the main modality map of the current frame group of multimodal data to be inspected in the current multimodal target time series data; Based on the main modality graph of the frame group multi-modal to-be-inspected data and the lesion detection box labeled on the main modality graph, the lesion feature description processing is performed on each lesion to determine the lesion region feature of each lesion, wherein the lesion region feature includes echo type, shape morphology and / or edge feature.

6. The method of claim 1 to 5, wherein, Load the generated lesion instance state information to the constructed diagnostic report generation model to generate the corresponding diagnostic report by using the diagnostic report generation model, wherein, When constructing the diagnostic report generation model, the following steps are included: A large language basic model is provided, and the large language basic model is fine-tuned by using the large language model training data set, wherein the QLoRA method is used for fine-tuning during fine-tuning; After training the large language basic model by using the large language model training data set to reach the target state, the diagnostic report generation model is formed based on the trained large language basic model; When generating the diagnostic report, the diagnostic report generation model processes the input lesion instance state information and generates the corresponding diagnostic report by combining the search enhancement.

7. The method of claim 6, wherein the method further comprises: When training the large language basic model, the speech model data samples in the large language model training data set are segmented and merged by using a segmenter to form speech model training samples; When fine-tuning in the QLoRA manner, the pre-training weights of the large language base model are frozen, and only learnable low-rank matrices are inserted in the key layers of the large language base model and the low-rank matrices to configure the query matrix and the value matrix as the multi-head attention mechanism within the large language base model.

8. The method of claim 1, wherein the method further comprises: The backbone network includes a shallow shared layer, a shallow backbone layer, a middle backbone layer, and a deep backbone layer connected in turn, wherein The shallow backbone layer, the middle backbone layer, and the deep backbone layer each include three backbone branches, and the corresponding three backbone branches in the shallow backbone layer, the middle backbone layer, and the deep backbone layer are connected to form three modal feature extraction branches; When performing multi-level feature extraction processing on each group of multi-modal to-be-inspected data, the to-be-inspected B modality image, the to-be-inspected blood flow modality image, and the to-be-inspected elasticity modality image are processed by the shallow shared layer, and then processed by the corresponding modal feature extraction branch, wherein The shallow B modality feature map, the shallow blood flow modality feature map, and the shallow elasticity modality feature map are output by the corresponding backbone branches of the shallow backbone layer, The middle B modality feature map, the middle blood flow modality feature map, and the middle elasticity modality feature map are output by the corresponding backbone branches of the middle backbone layer, The deep B modality feature map, the deep blood flow modality feature map, and the deep elasticity modality feature map are output by the corresponding backbone branches of the deep backbone layer.

9. The method of claim 8, wherein the method further comprises: The shallow shared layer includes at least two shallow shared convolution modules connected in series, wherein The two shallow shared convolution modules are used as the input layer and the output layer of the shallow shared layer; Each shallow shared convolution module includes a two-dimensional convolution layer, a batch normalization layer, and a shallow shared activation function layer connected in turn; The shallow shared activation function layer uses a SiLU activation function.

10. The method of claim 8, wherein the method further comprises: Each backbone branch of the shallow backbone layer includes a shallow first convolution-attention fusion module, a shallow convolution layer, and a shallow second convolution-attention fusion module connected in turn; Each backbone branch in the middle backbone layer includes a middle space-channel decoupling down-sampling module and a middle convolution-attention fusion module connected in turn; Each of the backbone branches in the deep layer trunk layer comprises, in sequence, a deep spatial-channel decoupling down-sampling module, a deep C2fCIB module, a feature pyramid pooling module, and a PSA module, wherein, The shallow first convolution-attention fusion module is adaptively connected with the shallow shared layer, the shallow second convolution-attention fusion module is adaptively connected with the middle layer spatial-channel decoupling down-sampling module, and the shallow second convolution-attention fusion modules of the three backbone branches in the shallow layer trunk layer respectively output a shallow B modality feature map, a shallow blood flow modality feature map, and a shallow elasticity modality feature map; The middle layer convolution-attention fusion module is adaptively connected with the deep layer spatial-channel decoupling down-sampling module, and the middle layer convolution-attention fusion modules of the three backbone branches in the middle layer trunk layer respectively output a middle layer B modality feature map, a middle layer blood flow modality feature map, and a middle layer elasticity modality feature map; The PSA modules of the three backbone branches in the deep layer trunk layer respectively output a deep layer B modality feature map, a deep layer blood flow modality feature map, and a deep layer elasticity modality feature map.

11. The method of claim 10, wherein the method further comprises: The shallow first convolution-attention fusion module, the shallow second convolution-attention fusion module, and the middle layer convolution-attention fusion module adopt the same structural form, comprising a cross-stage partial fusion first convolution block, wherein, The output ends of the cross-stage partial fusion first convolution block are respectively connected with a cross-stage first convolution self-attention module, a cross-stage second convolution self-attention module, and an input end of a cross-stage partial fusion splicer; The cross-stage first convolution self-attention module and the cross-stage second self-attention module are respectively connected with the corresponding input ends of the cross-stage partial fusion splicer, the output end of the cross-stage partial fusion splicer is connected with a cross-stage fusion second convolution block, and the cross-stage fusion second convolution block serves as an output layer.

12. The method of claim 1, wherein the method further comprises: The cross-modality attention module comprises three cross-modality attention units to perform cross-modality fusion on the shallow feature map group, the middle layer feature map group, and the deep layer feature map group respectively by using the three cross-modality attention units, wherein, When performing cross-modality fusion, the corresponding B modality feature map is taken as a main modality, and the corresponding blood flow modality feature map and the elasticity modality feature map are guided based on a cross-modality attention mechanism of a Transformer structure, wherein, When guiding the corresponding blood flow modality feature map and the elasticity modality feature map based on the cross-modality attention mechanism of the Transformer structure, the corresponding B modality feature map is taken as a Query source, and the corresponding blood flow modality feature map and the elasticity modality feature map are respectively taken as a Key source and a Value source.

13. The method of claim 1, wherein the method further comprises: The time sequence feature fusion module comprises three time sequence feature fusion units to perform time sequence fusion on all the shallow cross-modality fusion feature maps, the middle layer cross-modality fusion feature maps, and the deep layer cross-modality fusion feature maps respectively by using the three time sequence feature fusion units; When performing time sequence fusion, a global instance query on each cross-modality fusion feature map is determined, and each global instance query is decomposed into a frame-level feature by using a dynamic attention mechanism; All corresponding frame-level features of the same type cross-modality fusion feature map are spliced to form a corresponding frame-by-frame level splicing query; Each frame-level splicing query is weighted aggregated to form a corresponding time sequence fusion feature.

14. The method of claim 1 to 5, wherein, When generating the multi-modal data to be detected of each group, comprising: Obtaining a B-mode source image, a blood flow mode source image and an elasticity mode source image, and then respectively performing image preprocessing on the B-mode source image, the blood flow mode source image and the elasticity mode source image to respectively generate corresponding to-be-detected B-mode images, to-be-detected blood flow mode images and to-be-detected elasticity mode images after image preprocessing, wherein, When performing image preprocessing on the B-mode source image, at least wavelet threshold filtering processing is performed on the B-mode source image; When performing image preprocessing on the blood flow mode source image, the blood flow mode source image is first converted into an HSV color space, and after saturation channel feature enhancement is performed in the HSV color space, the blood flow mode source image is converted back into an RGB image; When performing image preprocessing on the elasticity mode source image, at least a pseudo-color image is mapped and converted into a single-channel grayscale image.

Images