A target classification method, device, electronic device and storage medium

Abstract

The present application provides a target classification method, device, electronic device and storage medium, the method comprising: obtaining an ultrasonic grayscale image and an ultrasonic contrast image collected for the same target object; extracting features from the ultrasonic grayscale image to obtain grayscale features, and extracting features from the ultrasonic contrast image to obtain contrast features; fusing the grayscale features and contrast features to obtain fused features; classifying the fused features to obtain the category of the target object. In the implementation of the above scheme, by fusing the grayscale features extracted from the ultrasonic grayscale image and the contrast features extracted from the ultrasonic contrast image collected for the same target object, fused features are obtained, and the fused features are classified, thereby effectively utilizing the complementary information between the grayscale features and the contrast features in different modalities, and effectively utilizing the diversity of image representation to improve the accuracy of image classification.

Description

Technical Field

[0001] The present application relates to the technical field of ultrasound image classification, and in particular to a target classification method, device, electronic device and storage medium. Background Art

[0002] At present, when an ultrasonic image is formed by reflecting ultrasonic waves from a target object, many situations may occur. For example, an ultrasonic grayscale image is formed by reflecting sound waves from a target object (such as a tissue or organ). The ultrasonic grayscale image cannot clearly show blood flow, blood perfusion, and vascular perfusion, etc., which makes it difficult to extract blood functional characteristics or hemodynamic characteristics. Therefore, the accuracy of classifying target objects based on ultrasonic grayscale images is not high. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a target classification method, device, electronic device and storage medium, which are used to improve the problem of low accuracy in classifying target objects.

[0004] The embodiment of the present application provides a target classification method, including: obtaining an ultrasonic grayscale image and an ultrasonic contrast image collected for the same target object; performing feature extraction on the ultrasonic grayscale image to obtain grayscale features, and performing feature extraction on the ultrasonic contrast image to obtain contrast features; performing feature fusion on the grayscale features and the contrast features to obtain fusion features; and classifying the fusion features to obtain the category of the target object. In the implementation process of the above scheme, by performing feature fusion on the grayscale features extracted from the ultrasonic grayscale image and the contrast features extracted from the ultrasonic contrast image collected for the same target object, fusion features are obtained, and the fusion features are classified, thereby effectively utilizing the complementary information between the grayscale features and the contrast features in different modalities, and effectively utilizing the diversity of image representation to improve the accuracy of image classification.

[0005] Optionally, in an embodiment of the present application, after acquiring the ultrasonic grayscale image and the ultrasonic contrast image collected for the same target object, it also includes: registering and aligning the ultrasonic grayscale image and the ultrasonic contrast image. In the implementation process of the above scheme, by registering and aligning the ultrasonic grayscale image and the ultrasonic contrast image, the structure of the same target object is better aligned in both the ultrasonic grayscale image and the ultrasonic contrast image, so that the complementary information and structural information of the target object in the ultrasonic grayscale image and the ultrasonic contrast image are easier to be identified and captured, and the effect of feature extraction and feature fusion of the two images is improved, the stability of feature extraction is increased, and the accuracy of image classification is further improved.

[0006] Optionally, in an embodiment of the present application, the fused features are classified, including: using a Transformer architecture model to classify the fused features. In the implementation process of the above scheme, the fused features are classified by using a Transformer architecture model. Since the Transformer architecture model has a self-attention mechanism and sequence modeling capabilities, it can better capture the long-range dependencies in the multimodal global information features, which is beneficial for the Transformer architecture model to understand the complex relationship between different regions in the multimodal image.

[0007] Optionally, in an embodiment of the present application, the Transformer architecture model includes: a classification layer; using the Transformer architecture model to classify the fused features, including: embedding the fused features to obtain embedded vector features; feature encoding the embedded vector features to obtain the encoded feature vectors; and using the classification layer to classify the encoded feature vectors. In the implementation of the above scheme, by feature encoding the embedded vector features and using the classification layer to classify the encoded feature vectors, since the Transformer architecture model can learn the complex features and correlations in the image, the classification layer of the Transformer architecture model can improve the classification accuracy of the fused features.

[0008] Optionally, in an embodiment of the present application, the Transformer architecture model further includes: a Patch embedding layer; embedding the fused features, including: obtaining a position encoding vector; and embedding the fused features and the position encoding vector using the Patch embedding layer. In the implementation of the above scheme, the fused features and the position encoding vector are embedded using the Patch embedding layer, thereby combining the embedding processing of the fused features and the position encoding vector, effectively capturing the long-range dependencies in the multimodal global information features, thereby facilitating the Transformer architecture model to understand the complex relationships between different regions in the multimodal image.

[0009] Optionally, in an embodiment of the present application, the Transformer architecture model further includes: a feature coding layer; and feature coding the embedded vector features, including: using the feature coding layer to feature code the embedded vector features. In the implementation of the above scheme, since the feature coding layer of the Transformer architecture model has a self-attention mechanism and sequence modeling capabilities, the feature coding layer of the Transformer architecture model can learn the complex features and correlations in the image, thereby improving the classification accuracy of the fused features.

[0010] Optionally, in an embodiment of the present application, the feature coding layer includes: a multi-head self-attention MSA layer and a multi-layer perceptron MLP layer; using the feature coding layer to feature encode the embedded vector features, including: using the MSA layer to process the embedded vector features to obtain attention features; using the MLP layer to feature encode the attention features. In the implementation process of the above scheme, the embedded vector features are processed by using the MSA layer to obtain the attention features, and the MLP layer is used to feature encode the attention features. Since the multi-head self-attention MSA layer and the multi-layer perceptron MLP layer have self-attention mechanisms and sequence modeling capabilities, the multi-head self-attention MSA layer and the multi-layer perceptron MLP layer can learn the complex features and correlations in the image, thereby improving the classification accuracy of the target object.

[0011] The embodiment of the present application also provides a target classification device, including: an ultrasound image acquisition module, used to acquire an ultrasound grayscale image and an ultrasound contrast image collected for the same target object; an image feature extraction module, used to perform feature extraction on the ultrasound grayscale image to obtain grayscale features, and to perform feature extraction on the ultrasound contrast image to obtain contrast features; a fusion feature acquisition module, used to perform feature fusion on the grayscale features and the contrast features to obtain fusion features; and a target category acquisition module, used to classify the fused features to obtain the category of the target object.

[0012] Optionally, in the embodiment of the present application, the target classification device further includes: an image configuration alignment module, which is used to align the ultrasound grayscale image and the ultrasound contrast image.

[0013] Optionally, in an embodiment of the present application, the target category acquisition module includes: a fusion feature classification submodule, which is used to classify the fusion features using a Transformer architecture model.

[0014] Optionally, in an embodiment of the present application, the Transformer architecture model includes: a classification layer; a fused feature classification submodule, including: an embedding vector acquisition unit, used to embed the fused features to obtain embedded vector features; a feature vector acquisition unit, used to feature encode the embedded vector features to obtain the encoded feature vector; and a feature vector classification unit, used to classify the encoded feature vector using the classification layer.

[0015] Optionally, in an embodiment of the present application, the Transformer architecture model also includes: a Patch embedding layer; an embedding vector acquisition unit, including: a position coding extraction subunit, for obtaining a position coding vector; and a feature vector processing subunit, for using the Patch embedding layer to embed the fused features and the position coding vector.

[0016] Optionally, in an embodiment of the present application, the Transformer architecture model further includes: a feature encoding layer; a feature vector acquisition unit, including: a vector feature encoding subunit, for performing feature encoding on embedded vector features using the feature encoding layer.

[0017] Optionally, in an embodiment of the present application, the feature encoding layer includes: a multi-head self-attention MSA layer and a multi-layer perceptron MLP layer; the vector feature encoding subunit includes: a feature processing subunit, used to use the MSA layer to process the embedded vector features to obtain attention features; a feature encoding subunit, used to use the MLP layer to feature encode the attention features.

[0018] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

[0019] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of a process flow of a target classification method provided by an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of the network architecture of the Transformer architecture model provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of the network architecture of the multi-head self-attention MSA layer provided in an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the structure of a target classification device provided in an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed, but merely represents the selected embodiments of the embodiments of the present application.

[0028] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0029] Before introducing the target classification method provided in the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0030] Convolutional Neural Networks (CNN) is a type of artificial neural network whose artificial neurons can respond to surrounding units and can process large images; a convolutional neural network can include convolution layers and pooling layers.

[0031] Feature extraction refers to using computers to extract image information and determine whether each point in an image can be used to extract certain features of an image. There are many specific ways to extract features, including those based on machine learning or neural networks.

[0032] It should be noted that the target classification method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and a non-x86 server includes a mainframe, a minicomputer and a UNIX server.

[0033] The following introduces application scenarios to which the target classification method is applicable. The application scenarios include but are not limited to: acquiring ultrasound grayscale images and ultrasound contrast images of target organs, and using the target classification method to fuse and classify features extracted from ultrasound grayscale images and ultrasound contrast images, etc., so as to effectively utilize complementary information between different image modalities of the target organ and the diversity of image representation to improve the accuracy of image classification.

[0034] See also Figure 1 The flowchart of the target classification method provided by the embodiment of the present application is shown; the embodiment of the present application provides a target classification method, including:

[0035] Step S110: Acquire an ultrasonic grayscale image and an ultrasonic contrast-enhanced image collected for the same target object.

[0036] The target object refers to the object for which the ultrasonic grayscale image and the ultrasonic contrast imaging image need to be acquired, such as an organ or tissue, etc. The organ here includes but is not limited to the kidney or the liver, etc.

[0037] Ultrasonic grayscale images are structural images formed by the reflection of ultrasound waves on organs or tissues. They are a common organ ultrasound modality image. Ultrasonic grayscale images can provide information such as the morphology, size, and structure of the organs.

[0038] Ultrasound contrast imaging is an image that is enhanced by adding ultrasound contrast agents to enhance the contrast and blood flow dynamics of the displayed image. Ultrasound contrast imaging is a type of ultrasound modality imaging with changed contrast. Ultrasound contrast imaging can clearly display blood flow dynamics and vascular perfusion.

[0039] Step S120: extracting features from the ultrasound grayscale image to obtain grayscale features, and extracting features from the ultrasound contrast imaging image to obtain contrast imaging features.

[0040] Step S130: performing feature fusion on the grayscale feature and the contrast feature to obtain a fused feature.

[0041] It is understandable that since grayscale features can better reflect information such as the morphology, size, and structure of organs, and contrast features can better reflect information such as blood flow and vascular perfusion, the feature fusion of grayscale features and contrast features can better combine the multimodal global information features of the two modalities, so that the fused features of the target object can provide more comprehensive and richer features, and better utilize the overall structure and features of the multimodal global information for classification.

[0042] Step S140: classify the fused features to obtain the category of the target object.

[0043] In the implementation process of the above scheme, the grayscale features extracted from the ultrasonic grayscale images and the contrast features extracted from the ultrasonic contrast images collected from the same target object are fused to obtain the fused features, and the fused features are classified, thereby effectively utilizing the complementary information between the grayscale features and the contrast features in different modalities, and effectively utilizing the diversity of image representation to improve the accuracy of image classification.

[0044] As an optional implementation of the above target classification method, after acquiring the ultrasound grayscale image and ultrasound contrast image collected for the same target object and before feature extraction, registration and alignment may be performed. This implementation may include:

[0045] Step S101: registering and aligning the ultrasound grayscale image and the ultrasound contrast imaging image.

[0046] It is understandable that the ultrasound grayscale image and ultrasound contrast image may have deviations in image features (such as shape or position, etc.) of the same target object (such as an organ) due to different environmental factors, and the environmental factors here include but are not limited to: imaging equipment, scanning batches and scanning parameters, etc. Therefore, in order to reduce the deviation caused by different environmental factors, the ultrasound grayscale image and the ultrasound contrast image can be registered and aligned.

[0047] The implementation method of the above step S101 is, for example: after acquiring the ultrasonic grayscale image and the ultrasonic contrast image collected for the same target object, the ultrasonic grayscale image and the ultrasonic contrast image are registered and aligned to obtain the registered and aligned ultrasonic grayscale image and the registered and aligned ultrasonic contrast image. Specifically, matching can be performed based on the intensity information of the image (for example, intensity mutual correlation information such as light intensity), for example: a sliding window is set in the ultrasonic grayscale image (as the main image) and the ultrasonic contrast image (as the reference image), and a cross-correlation registration algorithm is used to calculate the intensity mutual correlation information of the corresponding area of ​​the reference image according to the sliding window, and then the best matching position is found according to the intensity mutual correlation information. Among them, the cross-correlation registration algorithm can be implemented based on Python (Version 3.7.13) and OpenCV (Version 4.7.0.72), and the cv2.matchTemplate function is used to calculate the cross-correlation between the two images, and the matching position of the cross-correlation result is obtained by the cv2.minMaxLoc function, and the spatial transformation parameters related to the sliding window are determined according to the matching position. Then, feature extraction is performed on the registered and aligned ultrasound grayscale image to obtain grayscale features, and feature extraction is performed on the registered and aligned ultrasound contrast imaging image to obtain contrast imaging features.

[0048] In the implementation process of the above scheme, by aligning the ultrasonic grayscale image and the ultrasonic contrast image, the structure of the same target object is better aligned in both the ultrasonic grayscale image and the ultrasonic contrast image, so that the complementary information and structural information of the target object in the ultrasonic grayscale image and the ultrasonic contrast image are easier to be identified and captured, and the effect of feature extraction and feature fusion of the two images is improved, the stability of feature extraction is increased, and the accuracy of image classification is further improved.

[0049] In an optional implementation of the above step S120, a convolutional neural network (CNN) model can be used to extract features from the ultrasound grayscale image to obtain grayscale features, and the convolutional neural network model can be used to extract features from the ultrasound contrast image to obtain contrast features. For example, a neural network such as a LeNet network model, an AlexNet network model, a VGG network model, a GoogLeNet network model, and a ResNet network model can be used to extract features from the ultrasound grayscale image and the ultrasound contrast image; wherein the ResNet network model here is for example: ResNet22, ResNet38, ResNet50, ResNet101, and ResNet152. Optionally, each layer in the backbone layer of the above-mentioned ResNet network model (such as ResNet50) includes: a convolution layer, a Relu activation function layer, a batch normalization layer, and a residual module, etc.; in each residual module, a 1×1 convolution layer can be used for downsampling.

[0050] Optionally, before feature extraction is performed on the ultrasonic grayscale image and the ultrasonic contrast image, the ultrasonic grayscale image and the ultrasonic contrast image may be preprocessed, for example, the input ultrasonic grayscale image and the ultrasonic contrast image are compressed to a preset size (for example, 224×224), or semantic segmentation is performed on the ultrasonic grayscale image and the ultrasonic contrast image to segment out a regional image including the target object, so as to reduce the background area in the ultrasonic grayscale image and the ultrasonic contrast image, increase the features representing the effective information in the image, and improve the accuracy of image classification.

[0051] In an optional implementation of the above-mentioned step S130, a hierarchical feature fusion method can be adopted. For example, since the process of feature extraction of ultrasonic grayscale images and ultrasonic contrast images adopts the same CNN network architecture, the extracted grayscale ultrasonic image features and ultrasonic contrast image features can be hierarchically fused, so as to effectively integrate the features of the two modal images, the grayscale features extracted from the ultrasonic grayscale images and the contrast features extracted from the ultrasonic contrast images, into a unified multimodal feature vector, and finally obtain the fused features of the target object, so that the fused features of the target object can provide more comprehensive features and richer features, and better utilize the overall structure and features of the multimodal global information for classification.

[0052] In another optional implementation of the above step S130, a feature weighted fusion method can be used for cross-modal fusion. For example, a feature weighted fusion module (FWFM) including a fusion layer is used to cross-modally fuse the grayscale features extracted from the ultrasound grayscale image and the contrast features extracted from the ultrasound contrast image to obtain the fusion features of the target object; wherein the weight matrix of the above fusion layer is a trainable matrix, the initial weight values ​​of the multiple feature fusion layers in the above feature weighted fusion module can be set to 1, and the weight value of each feature fusion layer can be back propagated (Back Propagation) and updated together with the classification loss.

[0053] See also Figure 2 The network architecture diagram of the Transformer architecture model provided in the embodiment of the present application is shown; as an optional implementation of the above step S140, the implementation of classifying the fusion features may include:

[0054] Step S141: Use the Transformer architecture model to classify the fused features.

[0055] The Transformer architecture model refers to a neural network model that uses the Transformer architecture. The Transformer architecture model has achieved great success in the field of natural language processing (NLP). Since the Transformer architecture model has a self-attention mechanism and sequence modeling capabilities, the Transformer architecture model can learn the complex features and correlations in the image, thereby improving the classification accuracy of the fused features obtained by fusing grayscale features and contrast features.

[0056] Optionally, the above-mentioned fused features can be feature maps of a preset size (e.g., 224×224), or the fused features can be divided into a preset number of patches (e.g., 64), and each patch is projected into an embedded vector feature of a fixed length. In the specific implementation process, a special token can also be added to the input fixed-length vector, and the special token is used to characterize the category vector of the target object.

[0057] In the above implementation process, by using the Transformer architecture model to classify the fused features, the long-range dependencies in the multimodal global information features can be better captured, which is conducive to the Transformer architecture model to understand the complex relationship between different regions in the multimodal image.

[0058] As an optional implementation of the above step S141, the above Transformer architecture model may include: a Patch embedding layer and a classification layer; the above implementation of using the Transformer architecture model to classify the fused features may include:

[0059] Step S141a: Use the Patch embedding layer to embed the fused features to obtain embedded vector features.

[0060] An implementation of the above step S141a is, for example: using a Patch embedding layer to divide the fused feature (e.g., a 224×224 feature map) into a preset number of patches (e.g., 64 28×28 patches), and projecting each patch (Patch) into a fixed-length embedding vector feature of a D-dimensional embedding representation (a constant latent vector D is used on the Transformer layer).

[0061] Step S141b: perform feature encoding on the embedded vector feature to obtain an encoded feature vector.

[0062] Step S141c: Use the classification layer to classify the encoded feature vector.

[0063] An implementation of the above step S141c is, for example: using a softmax activation function layer as a classification layer, classifying the encoded feature vectors, and obtaining the category of the target object.

[0064] As an optional implementation of the above step S141, the above implementation of embedding the fusion feature using the patch embedding layer may include:

[0065] Step S141d: Obtain the position encoding vector.

[0066] The implementation method of the above step S114d includes: using the sine function and the cosine function to construct a position coding vector according to the input fusion feature, for example: using the sine function to calculate one of the feature vectors in the fusion feature to obtain the sine value, and using the cosine function to calculate one of the feature vectors in the fusion feature to obtain the cosine value, and then, splicing the sine value and the cosine value together to obtain the position coding vector.

[0067] Step S141e: Use the Patch embedding layer to embed the fused features and the position encoding vector to obtain the embedded vector features.

[0068] The implementation method of the above step S141e includes: using the Patch embedding layer to embed the fused features and the position encoding vector, for example: for each feature vector in the fused features, the feature vector is added to the position encoding vector to finally obtain the embedded vector feature.

[0069] As an optional implementation of the above step S141, the above Transformer architecture model may further include: a feature encoding layer; the above implementation of feature encoding of the embedded vector feature may include:

[0070] Step S141f: Use the feature encoding layer to feature encode the embedded vector features.

[0071] It is understandable that the above-mentioned feature encoding layer can adopt a Transformer encoder, which can include: a layer normalization (Layer Norm) module, a multi-head self-attention (MSA) layer, a multi-layer perceptron (MLP) layer and a residual connection layer, wherein the layer normalization module can be before the MSA layer and the MLP layer, and the residual module connection layer can be after the MSA layer and the MLP layer. In other words, the layer normalization module, the MSA layer, the MLP layer and the residual connection layer can be connected in sequence, and the specific structures of the above-mentioned MSA layer and the MLP layer will be described in detail below.

[0072] As an optional implementation of the above step S141f, the above feature encoding layer may include: a Transformer Encoder and a multi-layer perceptron MLP layer, and the Transformer Encoder may include: layer normalization (Layer Norm), a multi-head self-attention MSA layer and a multi-layer perceptron MLP layer, etc.; the above implementation of using the feature encoding layer to feature encode the embedded vector feature may include:

[0073] Step S141g: Use the MSA layer to process the embedded vector features to obtain attention features.

[0074] See also Figure 3The network architecture diagram of the multi-head self-attention MSA layer provided in the embodiment of the present application is shown; it can be understood that the above-mentioned multi-head self-attention (Multi-head Self-Attention, MSA) layer is also called the multi-head cross patch attention (Multi-head Cross Patch Attention) layer, and the MSA layer includes a spatial attention unit (Spatial Attention Unit) and a channel attention unit (Channel Attention Unit). Among them, the spatial attention unit includes: a point-by-point convolution layer (1×1Conv), a convolution layer (3×3Conv), a convolution layer (1×3Conv), a convolution layer (3×1Conv), and an activation layer (Activation). The spatial attention unit processes the input embedded vector features to obtain spatial features. The above-mentioned channel attention unit includes: a pooling layer (Pooling), a linear layer (Linear), and an activation layer (Activation). The channel attention unit can process the input spatial features to obtain channel features, and process the channel features with the features output by the pooling layer, and determine the processed output features as the above-mentioned attention features.

[0075] Step S141h: Use the MLP layer to feature encode the attention features.

[0076] In the implementation of the above-mentioned steps S141g and S141h, the above-mentioned MLP layer may include two fully connected layers with GELU nonlinearity, and these two fully connected layers can be used to feature encode the attention features.

[0077] See also Figure 4 The schematic diagram of the structure of the target classification device provided in the embodiment of the present application is shown; the embodiment of the present application provides a target classification device 200, including:

[0078] The ultrasound image acquisition module 210 is used to acquire an ultrasound grayscale image and an ultrasound contrast image collected for the same target object.

[0079] The image feature extraction module 220 is used to perform feature extraction on the ultrasound grayscale image to obtain grayscale features, and to perform feature extraction on the ultrasound contrast imaging image to obtain contrast imaging features.

[0080] The fusion feature acquisition module 230 is used to perform feature fusion on the grayscale feature and the contrast feature to obtain a fusion feature.

[0081] The target category acquisition module 240 is used to classify the fused features to obtain the category of the target object.

[0082] Optionally, in the embodiment of the present application, the target classification device further includes:

[0083] The image configuration alignment module is used to align the ultrasound grayscale image and the ultrasound contrast image.

[0084] Optionally, in the embodiment of the present application, the target category obtaining module includes:

[0085] The fused feature classification submodule is used to classify the fused features using the Transformer architecture model.

[0086] Optionally, in an embodiment of the present application, the Transformer architecture model includes: a classification layer; a fusion feature classification submodule, including:

[0087] The embedding vector obtaining unit is used to perform embedding processing on the fused features to obtain the embedding vector features.

[0088] The feature vector obtaining unit is used to perform feature encoding on the embedded vector feature to obtain the encoded feature vector.

[0089] The feature vector classification unit is used to classify the encoded feature vector using the classification layer.

[0090] Optionally, in the embodiment of the present application, the Transformer architecture model further includes: a Patch embedding layer; an embedding vector obtaining unit, including:

[0091] The position encoding extraction subunit is used to obtain the position encoding vector.

[0092] The feature vector processing subunit is used to embed the fused features and position encoding vector using the Patch embedding layer.

[0093] Optionally, in the embodiment of the present application, the Transformer architecture model further includes: a feature encoding layer; and a feature vector obtaining unit, including:

[0094] The vector feature encoding subunit is used to feature encode the embedded vector features using a feature encoding layer.

[0095] Optionally, in an embodiment of the present application, the feature encoding layer includes: a multi-head self-attention MSA layer and a multi-layer perceptron MLP layer; the vector feature encoding subunit includes:

[0096] The feature processing subunit is used to process the embedded vector features using the MSA layer to obtain the attention features.

[0097] Feature encoding subunit for feature encoding of attention features using MLP layers.

[0098] It should be understood that the device corresponds to the above-mentioned target classification method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be referred to in the above description, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0099] See also Figure 5 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0100] The embodiment of the present application also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by the processor 310 to execute the above method. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0101] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0102] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0103] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0104] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A target classification method, characterized in that: include: Acquire an ultrasonic grayscale image and an ultrasonic contrast-enhanced image collected for the same target object, and perform registration and alignment on the ultrasonic grayscale image and the ultrasonic contrast-enhanced image; Performing feature extraction on the ultrasound grayscale image to obtain grayscale features, and performing feature extraction on the ultrasound contrast imaging image to obtain contrast imaging features; Performing feature fusion on the grayscale feature and the contrast feature to obtain a fusion feature; Using the Transformer architecture model to classify the fused features, to obtain the category of the target object; Wherein, the registration and alignment of the ultrasonic grayscale image and the ultrasonic contrast image includes: taking the ultrasonic grayscale image as the main image and the ultrasonic contrast image as the reference image, then using a cross-correlation registration algorithm to calculate the intensity cross-correlation information of the corresponding area of ​​the reference image according to the sliding window set in the main image, and finding the best matching position according to the intensity cross-correlation information, and the best matching position determines the spatial transformation parameters of the sliding window, which are used to extract features from the ultrasonic grayscale image and the ultrasonic contrast image using the same CNN network architecture; The feature fusion of the grayscale feature and the contrast feature includes: performing hierarchical feature fusion on the grayscale feature and the contrast feature to obtain a multimodal feature vector, and the multimodal feature vector is used to obtain the fusion feature.

2. The method according to claim 1, characterized in that The Transformer architecture model includes: a classification layer; the classifying the fusion features using the Transformer architecture model includes: Embedding the fused features to obtain embedded vector features; Performing feature encoding on the embedded vector feature to obtain an encoded feature vector; The encoded feature vector is classified using the classification layer.

3. The method according to claim 2, characterized in that The Transformer architecture model further includes: a Patch embedding layer; and the embedding process of the fusion feature includes: Get the position encoding vector; The Patch embedding layer is used to embed the fused features and the position encoding vector.

4. The method according to claim 2, characterized in that The Transformer architecture model further includes: a feature encoding layer; the feature encoding of the embedded vector features includes: The feature encoding layer is used to perform feature encoding on the embedded vector features.

5. The method according to claim 4, characterized in that The feature encoding layer includes: a multi-head self-attention MSA layer and a multi-layer perceptron MLP layer; the feature encoding of the embedded vector feature using the feature encoding layer includes: Processing the embedding vector feature using the MSA layer to obtain an attention feature; The attention features are feature encoded using the MLP layer.

6. A target classification device, characterized in that: include: An ultrasonic image acquisition module, used to acquire an ultrasonic grayscale image and an ultrasonic contrast image collected for the same target object, and to register and align the ultrasonic grayscale image and the ultrasonic contrast image; An image feature extraction module, used to extract features from the ultrasound grayscale image to obtain grayscale features, and to extract features from the ultrasound contrast image to obtain contrast features; A fusion feature acquisition module, used for fusing the grayscale feature and the contrast feature to obtain a fusion feature; A target category acquisition module, used to classify the fused features using a Transformer architecture model to obtain the category of the target object; Wherein, the registration and alignment of the ultrasonic grayscale image and the ultrasonic contrast image includes: taking the ultrasonic grayscale image as the main image and the ultrasonic contrast image as the reference image, then using a cross-correlation registration algorithm to calculate the intensity cross-correlation information of the corresponding area of ​​the reference image according to the sliding window set in the main image, and finding the best matching position according to the intensity cross-correlation information, and the best matching position determines the spatial transformation parameters of the sliding window, which are used to extract features from the ultrasonic grayscale image and the ultrasonic contrast image using the same CNN network architecture; The feature fusion of the grayscale feature and the contrast feature includes: performing hierarchical feature fusion on the grayscale feature and the contrast feature to obtain a multimodal feature vector, and the multimodal feature vector is used to obtain the fusion feature.

7. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is performed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is executed.

Images