A method and device for identifying vascular lesions based on multimodal ultrasound images

Abstract

The present invention provides a method and device for identifying vascular lesions based on multimodal ultrasound images, which belongs to the field of image recognition technology. The method includes: extracting features from grayscale ultrasound images and color Doppler ultrasound images based on a basic feature extraction submodule to obtain multiple grayscale feature maps and multiple color feature maps; aligning the color feature map with the grayscale feature map based on a multimodal alignment submodule to obtain multiple alignment feature maps; fusing multiple alignment feature maps based on a multimodal fusion submodule to obtain a fusion feature map; and performing feature fusion and detection on the fusion feature map based on a feature fusion module and a detection module to obtain a recognition result. The present invention identifies vascular lesions based on ultrasound images of two different modes, thereby improving the recognition accuracy of vascular lesions.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a method and device for identifying vascular lesions based on multimodal ultrasonic images. Background Art

[0002] Infantile hemangioma and venous malformation are two common vascular diseases in infants. The existing technology relies on a single grayscale ultrasound image to detect and identify the area and type of vascular lesions.

[0003] However, in clinical practice, the two diseases have similar features in grayscale ultrasound images. When only grayscale ultrasound images are used to judge infantile hemangiomas and venous malformations, due to the limitation of the amount of information, "same disease, different images" and "same image, different disease" often occur during the examination, resulting in inaccurate identification results of vascular lesions.

[0004] Therefore, it is necessary to propose a method and device for vascular lesion recognition based on multimodal ultrasound images to improve the accuracy of vascular lesion recognition. Summary of the invention

[0005] In view of this, it is necessary to provide a method and device for identifying vascular lesions based on multimodal ultrasound images to solve the technical problem in the prior art that vascular lesions are identified by relying on a single grayscale ultrasound image, resulting in inaccurate identification results.

[0006] On the one hand, in order to solve the above technical problems, the present invention provides a vascular lesion recognition method based on multimodal ultrasound images, which is used to perform lesion recognition on multimodal ultrasound images based on a vascular lesion recognition model, wherein the vascular lesion recognition model includes a feature extraction module, a feature fusion module and a detection module, wherein the feature extraction module includes a basic feature extraction submodule, a multimodal alignment submodule and a multimodal fusion submodule; the multimodal ultrasound image includes a grayscale ultrasound image and a color Doppler ultrasound image;

[0007] The lesion recognition on the multimodal ultrasound image based on the vascular lesion recognition model includes:

[0008] Based on the basic feature extraction submodule, feature extraction is performed on the grayscale ultrasound image and the color Doppler ultrasound image respectively to obtain a plurality of grayscale feature maps and a plurality of color feature maps;

[0009] Aligning the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps;

[0010] Based on the multimodal fusion submodule, the multiple aligned feature maps are fused to obtain a fused feature map;

[0011] The fused feature map is sequentially subjected to feature fusion and detection based on the feature fusion module and the detection module to obtain a recognition result.

[0012] In a possible implementation, the multiple grayscale feature maps include a first grayscale feature map, a second grayscale feature map, a third grayscale feature map, and a fourth grayscale feature map, the multiple color feature maps include a first color feature map, a second color feature map, a third color feature map, and a fourth color feature map, and the multimodal alignment submodule includes a first multimodal alignment unit, a second multimodal alignment unit, and a third multimodal alignment unit;

[0013] Then, aligning the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps includes:

[0014] Aligning the first grayscale feature map, the second grayscale feature map, the third grayscale feature map, and the fourth grayscale feature map based on the first multimodal alignment unit to obtain a first alignment feature map;

[0015] Aligning the first color feature map, the second color feature map, the third color feature map, and the fourth color feature map based on the second multimodal alignment unit to obtain a second alignment feature map;

[0016] Based on the third multimodal alignment unit, the third grayscale feature map, the fourth grayscale feature map, the third color feature map, and the fourth color feature map are aligned to obtain a third alignment feature map.

[0017] In a possible implementation, the first multimodal alignment unit includes a separation convolution layer, a spatial convolution layer, and a bilinear interpolation layer connected in sequence, and the second multimodal alignment unit and the third multimodal alignment unit both include a separation convolution layer, a first spatial convolution layer, and an average pooling layer connected in sequence.

[0018] In a possible implementation, the multimodal fusion submodule includes a multimodal channel feature fusion unit and a three-channel feature encoding unit;

[0019] The multimodal channel feature fusion unit is used to adjust the number of channels of the first alignment feature map and the second alignment feature map to obtain a first adjustment feature map and a second adjustment feature map, and perform channel splicing on the first adjustment feature map, the second adjustment feature map and the third alignment feature to obtain a channel splicing map; the number of channels of the first adjustment feature map and the second adjustment feature map is the same as the number of channels of the third alignment feature map;

[0020] The three-channel feature encoding unit is used to encode the first aligned feature map, the second aligned feature map and the channel stitching map to obtain the fused feature map.

[0021] In a possible implementation, the multimodal channel feature fusion unit includes a multimodal average pooling layer, a maximum pooling layer, an overlay layer, an interpolation operation layer, and a first channel splicing layer;

[0022] The multimodal average pooling layer is used to perform an average pooling operation on the first aligned feature map to obtain an average pooling feature;

[0023] The maximum pooling layer is used to perform a maximum pooling operation on the first aligned feature map to obtain a maximum pooling feature;

[0024] The superposition layer is used to superimpose the average pooling feature and the maximum pooling feature to obtain the first adjusted feature map;

[0025] The interpolation operation layer is used to perform an interpolation operation on the second alignment feature map to obtain the second adjustment feature map;

[0026] The first channel stitching layer is used to perform channel stitching on the first adjustment feature map, the second adjustment feature map and the third alignment feature to obtain the channel stitching map.

[0027] In a possible implementation, the three-channel feature encoding unit includes a first channel convolution layer, a second channel convolution layer, a third channel convolution layer, a feature encoding layer, a fourth channel convolution layer, a fifth channel convolution layer, a sixth channel convolution layer, a second channel splicing layer, a batch normalization layer, and a first activation function layer;

[0028] The first channel convolution layer, the second channel convolution layer, and the third channel convolution layer are used to perform convolution operations on the first alignment feature map, the channel concatenation layer, and the second alignment feature map, respectively, to obtain a first convolution map, a second convolution map, and a third convolution map;

[0029] The feature encoding layer is used to perform encoding operations on the first convolution map, the second convolution map and the third convolution map to obtain a first encoding feature, a second encoding feature and a third encoding feature;

[0030] The fourth channel convolution layer, the fifth channel convolution layer, and the sixth channel convolution layer are used to perform convolution operations on the first coding feature, the second coding feature, and the third coding feature, respectively, to obtain a fourth convolution map, a fifth convolution map, and a sixth convolution map;

[0031] The second channel splicing layer is used to perform channel splicing on the fourth convolutional map, the fifth convolutional map and the sixth convolutional map to obtain a spliced ​​feature map;

[0032] The batch normalization layer and the first activation function layer sequentially perform batch normalization processing and activation processing on the spliced ​​feature map to obtain the fused feature map.

[0033] In a possible implementation, the basic feature extraction submodule includes a first basic feature extraction submodule for extracting features from the grayscale ultrasound image, and the first basic feature extraction submodule includes a CBS layer and four Vanilla module layers connected in sequence;

[0034] The CBS layer is used to perform preliminary feature extraction on the grayscale ultrasound image to obtain a preliminary grayscale feature map;

[0035] The four Vanilla module layers sequentially extract features from the grayscale preliminary feature map to obtain the multiple grayscale feature maps;

[0036] Each of the Vanilla module layers includes a first basic convolution layer, a second basic convolution layer, a third basic convolution layer, a fourth basic convolution layer and a fully connected layer which are connected in sequence.

[0037] In a possible implementation, the vascular lesion recognition model further includes a first large selective attention mechanism module, a second large selective attention mechanism module, and three third large selective attention mechanism modules;

[0038] The first large selective attention mechanism module is used to perform attention learning on the first aligned feature map and the fourth grayscale feature map to obtain a first attention feature map;

[0039] The second large selective attention mechanism module is used to perform attention learning on the second alignment feature map and the fourth color feature map to obtain a second attention feature map;

[0040] The third large-scale selective attention mechanism module is used to perform attention learning on the feature fusion results output by the feature fusion module to obtain a third attention feature map.

[0041] In a possible implementation, the first large selective attention mechanism module includes a decomposition layer, a first large convolution layer, a second large convolution layer, a third channel splicing layer, a pooling layer, a second spatial convolution layer, a first transition convolution layer, a second activation function layer, a first product layer, a second product layer, an attention overlay layer, a second transition convolution layer, and a third product layer connected in sequence;

[0042] The decomposition layer is used to decompose the first alignment feature map to obtain a first large core decomposition feature map and a second large core decomposition feature map;

[0043] The first large convolution layer and the second large convolution layer perform a first large convolution map and a second large convolution map on the first large kernel decomposition feature map and the second large kernel decomposition feature map respectively;

[0044] The third channel splicing layer is used to perform a channel splicing operation on the first large convolutional image and the second large convolutional image to obtain a large spliced ​​image;

[0045] The pooling layer, the second spatial convolution layer, the convolution layer and the second activation function layer sequentially perform pooling processing, spatial convolution processing, convolution processing and activation processing on the large mosaic image to obtain a first transition feature map and a second transition feature map;

[0046] The first product layer and the second product layer weight the first transition feature map and the second transition feature map by performing spatial selection masking respectively to obtain a first weighted feature map and a second weighted feature map;

[0047] The attention overlay layer is used to perform feature overlay on the first weighted feature map and the second weighted feature map to obtain an attention overlay map;

[0048] The second transition convolution layer is used to perform convolution processing on the attention superposition map to obtain an attention convolution feature map;

[0049] The third product layer is used to perform element-by-element multiplication of the attention feature map and the first alignment feature map to obtain the first attention feature map.

[0050] On the other hand, the present invention also provides a vascular lesion recognition device based on a multimodal ultrasound image, which is used to perform lesion recognition on a multimodal ultrasound image based on a vascular lesion recognition model, wherein the vascular lesion recognition model includes a feature extraction module, a feature fusion module and a detection module, wherein the feature extraction module includes a basic feature extraction submodule, a multimodal alignment submodule and a multimodal fusion submodule; the multimodal ultrasound image includes a grayscale ultrasound image and a color Doppler ultrasound image;

[0051] The vascular lesion recognition device based on multimodal ultrasound images comprises:

[0052] A feature extraction unit, configured to extract features from the grayscale ultrasound image and the color Doppler ultrasound image based on the basic feature extraction submodule to obtain a plurality of grayscale feature maps and a plurality of color feature maps;

[0053] A feature alignment unit, configured to align the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps;

[0054] A feature fusion unit, used for fusing the multiple aligned feature maps based on the multimodal fusion submodule to obtain a fused feature map;

[0055] The recognition unit is used to perform feature fusion and detection on the fused feature map based on the feature fusion module and the detection module in sequence to obtain a recognition result.

[0056] The beneficial effects of the present invention are as follows: the vascular lesion identification method based on multimodal ultrasound images provided by the present invention, by setting the multimodal ultrasound images to include grayscale ultrasound images and color Doppler ultrasound images, that is, the vascular lesions are identified based on two different modal ultrasound images, and the vascular lesion identification model can obtain the lesion tissue structure information from the grayscale ultrasound image, and the lesion blood flow information from the color Doppler ultrasound image, and these two different types of ultrasound images can provide different aspects of information and complement each other. Combining multimodal images can more accurately locate the lesion area, and comprehensive analysis of multimodal images can also help reduce the risk of misdiagnosis and missed diagnosis, and improve the accuracy of diagnosis.

[0057] Furthermore, considering that ultrasound images of different modalities may have different blur levels, angles, geometric shapes, etc., a multimodal alignment submodule is set to align the color feature map with the grayscale feature map to perform image registration and alignment before fusion. This can eliminate the adverse effects of differences in ultrasound images of different modalities on the recognition results, thereby further improving the recognition accuracy of vascular lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A schematic diagram of the structure of an embodiment of a vascular lesion recognition model provided by the present invention;

[0060] Figure 2 A schematic flow chart of an embodiment of a method for identifying vascular lesions based on multimodal ultrasound images provided by the present invention;

[0061] Figure 3 For the present invention Figure 2 A schematic flow chart of an embodiment of step S202;

[0062] Figure 4 A schematic diagram of the structure of an embodiment of a multimodal alignment submodule provided by the present invention;

[0063] Figure 5 A schematic diagram of the structure of an embodiment of a multimodal fusion submodule provided by the present invention;

[0064] Figure 6 A schematic diagram of the structure of an embodiment of a basic feature extraction submodule provided by the present invention;

[0065] Figure 7 A schematic diagram of the structure of an embodiment of the first large-scale selective attention mechanism module provided by the present invention;

[0066] Figure 8 A schematic diagram of the overall structure of the vascular lesion recognition model provided by the present invention;

[0067] Fig. 9 A schematic structural diagram of an embodiment of a vascular lesion identification device based on multimodal ultrasound images provided by the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] It should be understood that the schematic drawings are not drawn to scale. The flowchart used in the present invention shows the operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, under the guidance of the content of the present invention, can add one or more other operations to the flowchart, and can also remove one or more operations from the flowchart. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0070] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] The present invention provides a method and device for identifying vascular lesions based on multimodal ultrasound images, which are described below.

[0072] like Figure 1 As shown, the vascular lesion recognition model includes a feature extraction module, a feature fusion module and a detection module. The feature extraction module includes a basic feature extraction submodule, a multimodal alignment submodule and a multimodal fusion submodule; the multimodal ultrasound image includes a grayscale ultrasound image and a color Doppler ultrasound image;

[0073] Based on the above vascular lesion recognition model, Figure 2 As shown, a method for identifying vascular lesions based on multimodal ultrasound images includes:

[0074] S201, based on the basic feature extraction submodule, feature extraction is performed on the grayscale ultrasound image and the color Doppler ultrasound image respectively to obtain multiple grayscale feature maps and multiple color feature maps;

[0075] S202, aligning the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps;

[0076] S203, fusing multiple aligned feature maps based on the multimodal fusion submodule to obtain a fused feature map;

[0077] S204: performing feature fusion and detection on the fused feature map based on the feature fusion module and the detection module in sequence to obtain a recognition result.

[0078] Among them, the feature fusion module and detection module in the vascular lesion recognition model are the Neck and Head modules in YOLOv8, which will not be described here.

[0079] It should be understood that the identification result in step S204 includes the lesion area and the lesion type, and the lesion type is infantile hemangioma or venous malformation.

[0080] It should be noted that the vascular lesion recognition model needs to be trained in advance, specifically:

[0081] First, under the guidance of professional ultrasound doctors, 516 sets of infantile hemangioma multimodal data, 513 sets of venous malformation multimodal data, and 365 sets of normal ultrasound images were initially selected. The grayscale and color Doppler ultrasound image locations of each set corresponded. The inclusion and exclusion criteria for ultrasound images were as follows: Inclusion criteria: (1) Ultrasound examination; (2) Complete ultrasound data of the patient (grayscale and color Doppler ultrasound images corresponded); (3) Confirmed with infantile hemangioma; (4) Confirmed with venous malformation; (5) Normal ultrasound images were collected from the normal part of the patient; (6) Grayscale and color Doppler ultrasound images were clear, without motion blur or artifacts, which was conducive to analysis. Exclusion criteria: (1) Unqualified image quality; (2) Patients with incomplete information. According to the above criteria, 507 sets of infantile hemangioma and venous malformation multimodal ultrasound images and 355 sets of normal multimodal ultrasound images were retained.

[0082] Then, all multimodal ultrasound images were cropped to remove patient information, equipment information, useless background information and other areas in the ultrasound images, and the ultrasound parts of interest were retained; secondly, all ultrasound images were labeled under the guidance of ultrasound imaging experts. Labelme was used as the annotation software; thirdly, the multimodal ultrasound images of infantile hemangiomas and venous malformations were divided into 405 training sets, 51 validation sets and 51 test sets; the normal multimodal ultrasound image data were divided into 284 training sets, 35 validation sets and 36 test sets.

[0083] In order to facilitate the subsequent training of the network, the positions of the grayscale and color Doppler ultrasound images are corresponding, so the annotated parts of the grayscale and color Doppler ultrasound images are consistent.

[0084] Finally, due to the small number of normal multimodal ultrasound images, data augmentation was performed on the training set, validation set, and test set respectively. The enhancement method was randomly used by the algorithm using mirroring, horizontal flipping, and vertical flipping. The number of final data set divisions was the same as that of the other two diseases.

[0085] The initial model was trained and validated based on the training set and validation set to obtain a vascular lesion recognition model, and the test set was used to test the model performance.

[0086] Compared with the prior art, the vascular lesion identification method based on multimodal ultrasound images provided in the embodiment of the present invention, by setting the multimodal ultrasound image to include grayscale ultrasound image and color Doppler ultrasound image, that is, the vascular lesion is identified based on two different modal ultrasound images, and the vascular lesion identification model can obtain the lesion tissue structure information from the grayscale ultrasound image, and the lesion blood flow information from the color Doppler ultrasound image. These two different types of ultrasound images can provide different aspects of information and complement each other. Combining multimodal images can more accurately locate the lesion area, and comprehensive analysis of multimodal images can also help reduce the risk of misdiagnosis and missed diagnosis, and improve the accuracy of diagnosis.

[0087] Furthermore, taking into account that ultrasound images of different modalities may have different degrees of blur, angles, geometric shapes, etc., the embodiment of the present invention sets a multimodal alignment submodule to align the color feature map with the grayscale feature map to perform image registration and alignment before fusion, thereby eliminating the adverse effects of differences in ultrasound images of different modalities on the recognition results, and further improving the recognition accuracy of vascular lesions.

[0088] In a specific embodiment of the present invention, the plurality of grayscale feature maps include a first grayscale feature map, a second grayscale feature map, a third grayscale feature map, and a fourth grayscale feature map, and the plurality of color feature maps include a first color feature map, a second color feature map, a third color feature map, and a fourth color feature map.

[0089] To ensure accurate extraction of features of a single modality and between different modalities, in some embodiments of the present invention, Figure 1 As shown, the multimodal alignment submodule includes a first multimodal alignment unit, a second multimodal alignment unit and a third multimodal alignment unit;

[0090] If Figure 3 As shown, step S202 includes:

[0091] S301, aligning a first grayscale feature map, a second grayscale feature map, a third grayscale feature map, and a fourth grayscale feature map based on a first multimodal alignment unit to obtain a first alignment feature map;

[0092] S302, aligning the first color feature map, the second color feature map, the third color feature map, and the fourth color feature map based on the second multimodal alignment unit to obtain a second alignment feature map;

[0093] S303: align the third grayscale feature map, the fourth grayscale feature map, the third color feature map, and the fourth color feature map based on the third multimodal alignment unit to obtain a third aligned feature map.

[0094] The embodiment of the present invention aligns multiple grayscale feature maps of grayscale ultrasound images by setting a first multimodal alignment unit, that is, it ensures that the feature information inside the grayscale ultrasound image can be effectively aggregated. Similarly, by setting a second multimodal alignment unit, it ensures that the feature information inside the color Doppler ultrasound image can be effectively aggregated. Furthermore, the embodiment of the present invention extracts information between the grayscale ultrasound image and the color Doppler ultrasound image modalities by setting a third multimodal alignment unit, which can retain information that can interact between the two and ensure that the multimodal feature information is fully extracted, that is, the feature information inside the modality and the feature information between the modalities are extracted at the same time, thereby ensuring the accuracy of identifying vascular lesions.

[0095] In some embodiments of the present invention, Figure 4 As shown, the first multimodal alignment unit includes a separation convolution layer, a spatial convolution layer, and a bilinear interpolation layer connected in sequence, and the second multimodal alignment unit and the third multimodal alignment unit both include a separation convolution layer, a first spatial convolution layer, and an average pooling layer connected in sequence.

[0096] The embodiment of the present invention can enrich the receptive field of the multimodal features and pay more attention to the spatial context area of ​​the multimodal objects by setting the first multimodal alignment unit, the second multimodal alignment unit and the third multimodal alignment unit to include a separation convolution layer and a spatial convolution layer.

[0097] It should be noted that the multimodal alignment submodule selects the grayscale ultrasound image as the alignment template, and aligns the blood flow information in the color Doppler ultrasound image to the lesion in the grayscale ultrasound image to supplement the feature information.

[0098] Specifically, the calculation process within the mode (taking grayscale ultrasonic features as an example) and between modes is as shown in the following formula.

[0099]

[0100]

[0101] in Represents the calculation result of grayscale features, represents the calculation result of the characteristics between modes, represents a depth-wise separable convolution operation, represents the spatial convolution operation, and Represents the input feature image.

[0102] The embodiment of the present invention can map features to the minimum feature size through average pooling and bilinear interpolation operations. Since smaller feature sizes are calculated faster, the calculation delay time of feature fusion can be reduced, thereby improving the recognition efficiency of vascular lesions.

[0103] In some embodiments of the present invention, Figure 1 and Figure 5 As shown in the figure, the multimodal fusion submodule includes a multimodal channel feature fusion unit (Multimodal Channel Feature Fusion, MCFF) and a triple feature encoding unit (Triple Feature Encoding, TFE);

[0104] The multimodal channel feature fusion unit is used to adjust the number of channels of the first alignment feature map and the second alignment feature map to obtain the first adjustment feature map and the second adjustment feature map, and perform channel splicing on the first adjustment feature map, the second adjustment feature map and the third alignment feature to obtain a channel splicing map; the number of channels of the first adjustment feature map and the second adjustment feature map is the same as the number of channels of the third alignment feature map;

[0105] The three-channel feature encoding unit is used to encode the first aligned feature map, the second aligned feature map and the channel splicing map to obtain a fused feature map.

[0106] In a specific embodiment of the present invention, Figure 5 As shown, the multimodal channel feature fusion unit includes a multimodal average pooling layer, a maximum pooling layer, an overlay layer, an interpolation operation layer, and a first channel splicing layer;

[0107] The multimodal average pooling layer is used to perform an average pooling operation on the first aligned feature map to obtain an average pooling feature;

[0108] The maximum pooling layer is used to perform a maximum pooling operation on the first aligned feature map to obtain the maximum pooling feature;

[0109] The overlay layer is used to overlay the average pooling feature and the maximum pooling feature to obtain a first adjusted feature map;

[0110] The interpolation operation layer is used to perform an interpolation operation on the second alignment feature map to obtain a second adjustment feature map;

[0111] The first channel stitching layer is used to perform channel stitching on the first adjustment feature map, the second adjustment feature map and the third alignment feature map to obtain a channel stitching map.

[0112] The embodiment of the present invention sets adaptive maximum pooling and adaptive average pooling for the first alignment feature map, then adds them to obtain a processing result, performs an interpolation operation on the second alignment feature map, and finally splices the first adjustment feature map, the second adjustment feature map and the third alignment feature map having the same dimension in the channel dimension. Based on cross-channel interaction, important structural features in the learning positioning image can be ensured to ensure the accuracy of recognition.

[0113] In a specific embodiment of the present invention, Figure 5 As shown, the three-channel feature encoding unit includes a first channel convolution layer, a second channel convolution layer, a third channel convolution layer, a feature encoding layer, a fourth channel convolution layer, a fifth channel convolution layer, a sixth channel convolution layer, a second channel concatenation layer, a batch normalization layer, and a first activation function layer;

[0114] The first channel convolution layer, the second channel convolution layer and the third channel convolution layer are used to perform convolution operations on the first alignment feature map, the channel concatenation layer and the second alignment feature map, respectively, to obtain the first convolution map, the second convolution map and the third convolution map;

[0115] The feature encoding layer is used to perform encoding operations on the first convolutional map, the second convolutional map, and the third convolutional map to obtain a first encoding feature, a second encoding feature, and a third encoding feature;

[0116] The fourth channel convolution layer, the fifth channel convolution layer and the sixth channel convolution layer are used to perform convolution operations on the first coding feature, the second coding feature and the third coding feature, respectively, to obtain a fourth convolution map, a fifth convolution map and a sixth convolution map;

[0117] The second channel splicing layer is used to perform channel splicing on the fourth convolution map, the fifth convolution map and the sixth convolution map to obtain a spliced ​​feature map;

[0118] The batch normalization layer and the first activation function layer perform batch normalization and activation processing on the concatenated feature map in turn to obtain a fused feature map.

[0119] Among them, the function of the three-channel feature encoding unit is to split or add one of the modal features to the features of another modality, and interactively fuse the feature information to improve the feature fusion effect. The interactive fusion between different modal features can be realized through the feature encoding layer, which can further improve the accuracy of multimodal feature extraction.

[0120] Among them, the activation function of the first activation function layer is the SiLU activation function.

[0121] In order to improve the recognition efficiency of vascular lesions while ensuring the accuracy of vascular lesion recognition, in some embodiments of the present invention, Figure 6 As shown, the basic feature extraction submodule includes a first basic feature extraction submodule for extracting features from a grayscale ultrasound image, and the first basic feature extraction submodule includes a CBS layer and four Vanilla module layers connected in sequence;

[0122] The CBS layer is used to perform preliminary feature extraction on the grayscale ultrasound image to obtain a preliminary grayscale feature map;

[0123] The four Vanilla module layers extract features from the grayscale preliminary feature map in turn to obtain multiple grayscale feature maps;

[0124] Each Vanilla module layer includes a first basic convolution layer, a second basic convolution layer, a third basic convolution layer, a fourth basic convolution layer and a fully connected layer which are connected in sequence.

[0125] The embodiment of the present invention sets the Vanilla module layer as the feature extraction module layer. Since the Vanilla module layer only includes basic base convolution layers and fully connected layers, without complex connections or jump connections, the calculation amount and parameter amount of the model are reduced, which can effectively simplify the vascular lesion recognition model and thus improve the recognition efficiency.

[0126] Among them, the kernel size of the first basic convolution layer, the second basic convolution layer, the third basic convolution layer, and the fourth basic convolution layer is 1×1, which can use the minimum computational cost for each layer while retaining the feature map information.

[0127] It should be noted that the CBS layer consists of a convolutional layer, a batch normalization layer, and an activation function layer whose activation function is SiLU, which are connected in sequence.

[0128] It should be understood that the basic feature extraction submodule includes a second basic feature extraction submodule for extracting features from Doppler color ultrasound images, and its structure is the same as that of the first basic feature extraction submodule, which will not be described in detail here.

[0129] Since the lesion area in the multimodal ultrasound image is usually small, there is a lot of redundant information in the entire image. In order to reduce the influence of the redundant information on the recognition result, in some embodiments of the present invention, Figure 1 As shown, the vascular lesion recognition model also includes a first large selective attention mechanism module, a second large selective attention mechanism module, and three third large selective attention mechanism modules;

[0130] The first large selective attention mechanism module is used to perform attention learning on the first alignment feature map and the fourth grayscale feature map to obtain a first attention feature map;

[0131] The second large selective attention mechanism module is used to perform attention learning on the second alignment feature map and the fourth color feature map to obtain a second attention feature map;

[0132] The third large selective attention mechanism module is used to perform attention learning on the feature fusion results output by the feature fusion module to obtain the third attention feature map.

[0133] The embodiment of the present invention sets a first large selective attention mechanism module, a second large selective attention mechanism module and three third large selective attention mechanism modules, and its attention mechanism is adaptive and can automatically adjust the distribution of attention according to the image content, so that the model can flexibly adapt to different image features and detection tasks. In addition, the large selective attention mechanism (LSK) module helps to effectively combine local information with global information to better understand the context of the entire image. Finally, the introduction of this mechanism can improve the robustness of the model to various conditions and changes, making it more suitable for the detection of grayscale ultrasound images and different lesion information. In summary, the LSK module helps to optimize the grayscale ultrasound image detection task of medical images, improve the model performance and sensitivity to key information. Thereby, the accuracy of vascular lesion identification can be further improved.

[0134] It should be noted that the structure of the LSK module is similar to the conventional SK module, but there are two key differences between the two. First, the LSK module relies more on the sequence of large kernels through decomposition, which is different from most existing attention-based methods and does not lose image information. Second, the SK module only aggregates the information of the large kernel in the channel dimension, while the LSK module can adaptively aggregate the information of the large kernel in the spatial dimension. Because channel selection cannot simulate the spatial variance of different lesion sizes in the image space, this design is more intuitive and effective for global learning of lesion features.

[0135] It should be understood that the module structures of the first large-scale selective attention mechanism module, the second large-scale selective attention mechanism module and the third large-scale selective attention mechanism module are the same, and the first large-scale selective attention mechanism module is used as an example for explanation. Figure 7 As shown, the first large selective attention mechanism module includes a decomposition layer, a first large convolution layer, a second large convolution layer, a third channel splicing layer, a pooling layer, a second spatial convolution layer, a first transition convolution layer, a second activation function layer, a first product layer, a second product layer, an attention overlay layer, a second transition convolution layer and a third product layer connected in sequence;

[0136] The decomposition layer is used to decompose the first alignment feature map to obtain a first large core decomposition feature map and a second large core decomposition feature map;

[0137] The first large convolution layer and the second large convolution layer perform a first large convolution map and a second large convolution map on the first large kernel decomposition feature map and the second large kernel decomposition feature map respectively;

[0138] The third channel splicing layer is used to perform channel splicing operation on the first large convolutional map and the second large convolutional map to obtain a large spliced ​​map;

[0139] The pooling layer, the second spatial convolution layer, the convolution layer, and the second activation function layer sequentially perform pooling processing, spatial convolution processing, convolution processing, and activation processing on the large spliced ​​image to obtain a first transition feature map and a second transition feature map;

[0140] The first product layer and the second product layer weight the first transition feature map and the second transition feature map by performing spatial selection masking respectively to obtain a first weighted feature map and a second weighted feature map;

[0141] The attention overlay layer is used to perform feature overlay on the first weighted feature map and the second weighted feature map to obtain an attention overlay map;

[0142] The second transition convolution layer is used to perform convolution processing on the attention superposition map to obtain the attention convolution feature map;

[0143] The third product layer is used to perform element-by-element multiplication of the attention feature map and the first alignment feature map to obtain the first attention feature map.

[0144] It should be noted that: since the first large selective attention mechanism module is used to learn attention on the first aligned feature map and the fourth grayscale feature map, the first large core decomposition feature map after decomposition by the decomposition layer is the first aligned feature map, and the second large core decomposition feature map is the fourth grayscale feature map. In other words, when there are two inputs to the first large selective attention mechanism module, no additional decomposition is required, and the two inputs can be directly used as the two large decomposition feature maps after decomposition.

[0145] Combining all the above mold layer structures, we can get Figure 8 , Figure 8 This is a schematic diagram of the overall structure of the vascular lesion recognition model proposed in an embodiment of the present invention, wherein GUS is a grayscale ultrasound image, CDUS is a color Doppler ultrasound image, Backbone is a feature extraction module, Neck is a feature fusion module, Head is a detection module, MFA is a multimodal feature alignment submodule; MFF is a multimodal fusion submodule, and LSK is a large selective attention mechanism module.

[0146] In order to verify the superiority of the vascular lesion recognition method based on multimodal ultrasound images proposed in the embodiment of the present invention, the embodiment of the present invention also conducted comparative verification on three aspects of model lightweighting, namely, introducing the Vanilla module layer, introducing a large selective attention mechanism module, and multimodality, as follows:

[0147] 1. Lightweight network comparison experimental results

[0148] The following evaluation indicators are used in the embodiment of the present invention: F1 Score, mAP50, mAP50 and GFLOPs, which measures the computational complexity of the model. F1 Score is the harmonic mean of precision and recall. mAP50 and mAP50-95 represent the average accuracy when the IoU threshold is 0.5 and the average accuracy when the IoU threshold changes from 0.5 to 0.95, respectively.

[0149] In order to verify the effectiveness of the lightweight network used in the embodiment of the present invention, the performance of the lightweight network used in the embodiment of the present invention is compared with other lightweight networks currently available using the grayscale ultrasound image part of the multimodal ultrasound image dataset under the same experimental environment. Table 1 compares the overall performance of the performance of all categories after replacing the backbone feature extraction network of YOLOv8 with MobileNet v3, ShuffleNet v2, EfficientNet and Vanilla module layers. GFLOPS represents the performance of model calculation, and the computing cost and computing power are proportional to the value of GFLOPS. From the perspective of the lightweight network, the GFLOPS of the Vanilla module layer is lower than that of other networks, which is reduced by about 31%, about 32.6% and about 34.6% respectively. Therefore, although the Vanilla module layer may have low computational efficiency in some cases, considering the balance of performance, choosing the Vanilla module layer is a choice that takes into account both performance and computational efficiency. From the evaluation parameter results of the model, the performance of the Vanilla module layer in various categories is relatively balanced. Although it may be slightly lower than a specific model in some cases, overall, the module can provide reasonable performance.

[0150] Table 1 Evaluation parameters of lightweight network comparison experiment

[0151]

[0152] 2. Comparative Experimental Results of Large-Scale Selective Attention Mechanism

[0153] In order to verify the effectiveness of the large selective attention mechanism module used in the embodiment of the present invention, the performance of the LSK module used in this article is compared with that of the SK module in the grayscale ultrasound image part of the multimodal ultrasound image dataset under the same experimental environment. Table 2 compares the performance of the SK module and the LSK module added to YOLOv8 in terms of overall performance and single category. All in the table represents all datasets, IH in the table represents infantile hemangioma, VM represents venous malformation, and Normal represents a dataset without lesions. First, the overall performance of the two is compared. LSK is 2.6% and 1% higher in F1 score and mAP50-95, respectively. In addition, in terms of computational complexity (GFLOPS), the LSK model shows a relatively low computational load in each category, which indicates that the LSK model has higher computational efficiency and lower computational cost than the SK model while improving detection performance.

[0154] Overall, the LSK model shows significant performance improvements in all categories compared to the SK model, especially in mAP50-95. In addition, the relatively low computational complexity of the LSK model also makes it a favorable choice in practical applications.

[0155] Table 2 Evaluation parameters of the selective attention mechanism comparison experiment

[0156]

[0157] 3. Comparative experiment of multimodal ultrasound image detection

[0158] In order to evaluate the advancement and effectiveness of the multimodal ultrasound image target detection algorithm, under the same experimental environment and training conditions, this paper modified YOLOv8 into a multimodal input as a comparison model.

[0159] Table 3 compares the performance of the proposed algorithm and multimodal input YOLOv8. The YOLOv8 model is changed to multimodal input, but the multimodal alignment submodule and multimodal fusion submodule are not used. From the perspective of the overall category performance, the model (MUS-YOLO) that changes the model input to multimodal input and uses the multimodal alignment submodule and multimodal fusion submodule has improved F1 score, mAP50 and mAP50-95 by 2%, 1.5% and 7.2% respectively compared to YOLOv8 that only changes to dual input. In the IH category, the improvements are 2.9%, 2.8% and 5.9% respectively, in the VM category, the F1 score and mAP50-95 are improved by 2.1% and 4.9% respectively, and in the Normal category, the improvements are 0.9%, 2.3% and 10.7% respectively. In terms of computational complexity, the computational cost is effectively reduced by about 50% compared to YOLOv8. This shows that the vascular lesion recognition model in the embodiment of the present invention can achieve excellent performance in target detection tasks with a lighter computational burden, providing a more efficient solution for environments with limited computing resources.

[0160] Table 3 Evaluation parameters of multimodal detection algorithm comparison experiment

[0161]

[0162] In summary, the vascular lesion recognition method based on multimodal ultrasound images proposed in the embodiment of the present invention is superior and practical in vascular lesion recognition compared with the methods in the prior art.

[0163] In order to better implement the vascular lesion recognition method based on multimodal ultrasound images in the embodiment of the present invention, based on the vascular lesion recognition method based on multimodal ultrasound images, the embodiment of the present invention also provides a vascular lesion recognition device based on multimodal ultrasound images, which is used to perform lesion recognition on multimodal ultrasound images based on a vascular lesion recognition model. The vascular lesion recognition model includes a feature extraction module, a feature fusion module and a detection module. The feature extraction module includes a basic feature extraction submodule, a multimodal alignment submodule and a multimodal fusion submodule; the multimodal ultrasound image includes a grayscale ultrasound image and a color Doppler ultrasound image; Fig. 9 As shown, a vascular lesion identification device 900 based on multimodal ultrasound images includes:

[0164] A feature extraction unit 901 is used to extract features from the grayscale ultrasound image and the color Doppler ultrasound image based on the basic feature extraction submodule to obtain a plurality of grayscale feature maps and a plurality of color feature maps;

[0165] A feature alignment unit 902 is used to align the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps;

[0166] A feature fusion unit 903 is used to fuse multiple aligned feature maps based on the multimodal fusion submodule to obtain a fused feature map;

[0167] The recognition unit 904 is used to perform feature fusion and detection on the fused feature map based on the feature fusion module and the detection module in sequence to obtain a recognition result.

[0168] The vascular lesion identification device 900 based on multimodal ultrasound images provided in the above-mentioned embodiment can implement the technical solution described in the above-mentioned embodiment of the vascular lesion identification method based on multimodal ultrasound images. The specific implementation principles of the above-mentioned modules or units can refer to the corresponding contents in the above-mentioned embodiment of the vascular lesion identification method based on multimodal ultrasound images, which will not be repeated here.

[0169] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0170] The above is a detailed introduction to the method and device for identifying vascular lesions based on multimodal ultrasound images provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for identifying vascular lesions based on multimodal ultrasound images, characterized in that: Used to perform lesion recognition on a multimodal ultrasound image based on a vascular lesion recognition model, wherein the vascular lesion recognition model includes a feature extraction module, a feature fusion module and a detection module, wherein the feature extraction module includes a basic feature extraction submodule, a multimodal alignment submodule and a multimodal fusion submodule; the multimodal ultrasound image includes a grayscale ultrasound image and a color Doppler ultrasound image; The lesion recognition on the multimodal ultrasound image based on the vascular lesion recognition model includes: Based on the basic feature extraction submodule, feature extraction is performed on the grayscale ultrasound image and the color Doppler ultrasound image respectively to obtain a plurality of grayscale feature maps and a plurality of color feature maps; Aligning the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps; Based on the multimodal fusion submodule, the multiple aligned feature maps are fused to obtain a fused feature map; Sequentially performing feature fusion and detection on the fused feature map based on the feature fusion module and the detection module to obtain a recognition result; The basic feature extraction submodule includes a first basic feature extraction submodule for extracting features from the grayscale ultrasound image, wherein the first basic feature extraction submodule includes a CBS layer and four Vanilla module layers connected in sequence; The CBS layer is used to perform preliminary feature extraction on the grayscale ultrasound image to obtain a preliminary grayscale feature map; The four Vanilla module layers sequentially extract features from the grayscale preliminary feature map to obtain the multiple grayscale feature maps; Each of the Vanilla module layers includes a first basic convolution layer, a second basic convolution layer, a third basic convolution layer, a fourth basic convolution layer and a fully connected layer which are connected in sequence.

2. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 1, characterized in that: The plurality of grayscale feature maps include a first grayscale feature map, a second grayscale feature map, a third grayscale feature map, and a fourth grayscale feature map, the plurality of color feature maps include a first color feature map, a second color feature map, a third color feature map, and a fourth color feature map, and the multimodal alignment submodule includes a first multimodal alignment unit, a second multimodal alignment unit, and a third multimodal alignment unit; Then, aligning the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps includes: Aligning the first grayscale feature map, the second grayscale feature map, the third grayscale feature map, and the fourth grayscale feature map based on the first multimodal alignment unit to obtain a first alignment feature map; Aligning the first color feature map, the second color feature map, the third color feature map, and the fourth color feature map based on the second multimodal alignment unit to obtain a second alignment feature map; Based on the third multimodal alignment unit, the third grayscale feature map, the fourth grayscale feature map, the third color feature map, and the fourth color feature map are aligned to obtain a third alignment feature map.

3. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 2, characterized in that: The first multimodal alignment unit includes a separation convolution layer, a spatial convolution layer and a bilinear interpolation layer connected in sequence, and the second multimodal alignment unit and the third multimodal alignment unit both include a separation convolution layer, a first spatial convolution layer and an average pooling layer connected in sequence.

4. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 2, characterized in that: The multimodal fusion submodule includes a multimodal channel feature fusion unit and a three-channel feature encoding unit; The multimodal channel feature fusion unit is used to adjust the number of channels of the first alignment feature map and the second alignment feature map to obtain a first adjusted feature map and a second adjusted feature map, and perform channel splicing on the first adjusted feature map, the second adjusted feature map and the third alignment feature map to obtain a channel splicing map; the number of channels of the first adjusted feature map and the second adjusted feature map is the same as the number of channels of the third alignment feature map; The three-channel feature encoding unit is used to encode the first aligned feature map, the second aligned feature map and the channel stitching map to obtain the fused feature map.

5. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 4, characterized in that: The multimodal channel feature fusion unit includes a multimodal average pooling layer, a maximum pooling layer, an overlay layer, an interpolation operation layer and a first channel splicing layer; The multimodal average pooling layer is used to perform an average pooling operation on the first aligned feature map to obtain an average pooling feature; The maximum pooling layer is used to perform a maximum pooling operation on the first aligned feature map to obtain a maximum pooling feature; The superposition layer is used to superimpose the average pooling feature and the maximum pooling feature to obtain the first adjusted feature map; The interpolation operation layer is used to perform an interpolation operation on the second alignment feature map to obtain the second adjustment feature map; The first channel stitching layer is used to perform channel stitching on the first adjustment feature map, the second adjustment feature map and the third alignment feature map to obtain the channel stitching map.

6. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 4, characterized in that: The three-channel feature encoding unit includes a first channel convolution layer, a second channel convolution layer, a third channel convolution layer, a feature encoding layer, a fourth channel convolution layer, a fifth channel convolution layer, a sixth channel convolution layer, a second channel splicing layer, a batch normalization layer and a first activation function layer; The first channel convolution layer, the second channel convolution layer, and the third channel convolution layer are used to perform convolution operations on the first alignment feature map, the channel concatenation layer, and the second alignment feature map, respectively, to obtain a first convolution map, a second convolution map, and a third convolution map; The feature encoding layer is used to perform encoding operations on the first convolution map, the second convolution map and the third convolution map to obtain a first encoding feature, a second encoding feature and a third encoding feature; The fourth channel convolution layer, the fifth channel convolution layer, and the sixth channel convolution layer are used to perform convolution operations on the first coding feature, the second coding feature, and the third coding feature, respectively, to obtain a fourth convolution map, a fifth convolution map, and a sixth convolution map; The second channel splicing layer is used to perform channel splicing on the fourth convolutional map, the fifth convolutional map and the sixth convolutional map to obtain a spliced ​​feature map; The batch normalization layer and the first activation function layer sequentially perform batch normalization processing and activation processing on the spliced ​​feature map to obtain the fused feature map.

7. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 2, characterized in that: The vascular lesion recognition model also includes a first large selective attention mechanism module, a second large selective attention mechanism module and three third large selective attention mechanism modules; The first large selective attention mechanism module is used to perform attention learning on the first aligned feature map and the fourth grayscale feature map to obtain a first attention feature map; The second large selective attention mechanism module is used to perform attention learning on the second alignment feature map and the fourth color feature map to obtain a second attention feature map; The third large-scale selective attention mechanism module is used to perform attention learning on the feature fusion results output by the feature fusion module to obtain a third attention feature map.

8. The method for identifying vascular lesions based on multimodal ultrasound images according to claim 7, characterized in that: The first large selective attention mechanism module includes a decomposition layer, a first large convolution layer, a second large convolution layer, a third channel splicing layer, a pooling layer, a second spatial convolution layer, a first transition convolution layer, a second activation function layer, a first product layer, a second product layer, an attention overlay layer, a second transition convolution layer and a third product layer connected in sequence; The decomposition layer is used to decompose the first alignment feature map to obtain a first large core decomposition feature map and a second large core decomposition feature map; The first large convolution layer and the second large convolution layer perform a first large convolution map and a second large convolution map on the first large kernel decomposition feature map and the second large kernel decomposition feature map respectively; The third channel splicing layer is used to perform a channel splicing operation on the first large convolutional image and the second large convolutional image to obtain a large spliced ​​image; The pooling layer, the second spatial convolution layer, the convolution layer and the second activation function layer sequentially perform pooling processing, spatial convolution processing, convolution processing and activation processing on the large mosaic image to obtain a first transition feature map and a second transition feature map; The first product layer and the second product layer weight the first transition feature map and the second transition feature map by performing spatial selection masking respectively to obtain a first weighted feature map and a second weighted feature map; The attention overlay layer is used to perform feature overlay on the first weighted feature map and the second weighted feature map to obtain an attention overlay map; The second transition convolution layer is used to perform convolution processing on the attention superposition map to obtain an attention convolution feature map; The third product layer is used to perform element-by-element multiplication of the attention feature map and the first alignment feature map to obtain the first attention feature map.

9. A vascular lesion identification device based on multimodal ultrasound images, characterized in that: Used to perform lesion recognition on a multimodal ultrasound image based on a vascular lesion recognition model, wherein the vascular lesion recognition model includes a feature extraction module, a feature fusion module and a detection module, wherein the feature extraction module includes a basic feature extraction submodule, a multimodal alignment submodule and a multimodal fusion submodule; the multimodal ultrasound image includes a grayscale ultrasound image and a color Doppler ultrasound image; The vascular lesion recognition device based on multimodal ultrasound images comprises: A feature extraction unit, configured to extract features from the grayscale ultrasound image and the color Doppler ultrasound image based on the basic feature extraction submodule to obtain a plurality of grayscale feature maps and a plurality of color feature maps; A feature alignment unit, configured to align the color feature map with the grayscale feature map based on the multimodal alignment submodule to obtain a plurality of aligned feature maps; A feature fusion unit, used for fusing the multiple aligned feature maps based on the multimodal fusion submodule to obtain a fused feature map; A recognition unit, used to perform feature fusion and detection on the fused feature map based on the feature fusion module and the detection module in sequence to obtain a recognition result; The basic feature extraction submodule includes a first basic feature extraction submodule for extracting features from the grayscale ultrasound image, wherein the first basic feature extraction submodule includes a CBS layer and four Vanilla module layers connected in sequence; The CBS layer is used to perform preliminary feature extraction on the grayscale ultrasound image to obtain a preliminary grayscale feature map; The four Vanilla module layers sequentially extract features from the grayscale preliminary feature map to obtain the multiple grayscale feature maps; Each of the Vanilla module layers includes a first basic convolution layer, a second basic convolution layer, a third basic convolution layer, a fourth basic convolution layer and a fully connected layer which are connected in sequence.

Images