Pulmonary artery image anomaly classification method and device, equipment and medium

Abstract

The embodiment of the invention provides a pulmonary artery image anomaly classification method and device, equipment and a medium. The pulmonary artery image anomaly classification method comprises the steps that a target CT image and a target ultrasonic image of a pulmonary artery are acquired; respectively coding the target CT image and the target ultrasonic image, and respectively extracting first global and local features of the target CT image and second global and local features of the target ultrasonic image after coding; performing weighted fusion on the first global and local features and the second global and local features through a preset multi-head attention mechanism to obtain multi-modal fusion features; performing decoding processing on the multi-modal fusion features, and performing reconstruction after decoding processing to obtain a target fusion image of the pulmonary artery; and performing anomaly classification based on the target fusion image to obtain an anomaly classification result of the pulmonary artery, thereby improving the accuracy of anomaly classification of the pulmonary artery image.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, and in particular to a method, apparatus, device and medium for abnormal classification of pulmonary artery images. Background Technology

[0002] The pulmonary artery is a blood vessel that connects the right ventricle of the heart to the lungs. It is responsible for transporting hypoxic venous blood from the heart to the lungs for gas exchange. Pulmonary hypertension is a complex and serious cardiopulmonary vascular disease characterized by elevated pulmonary artery pressure and increased pulmonary vascular resistance, which can eventually lead to right heart failure and even death.

[0003] With the development of artificial intelligence technology, neural networks can now be used to classify pulmonary artery images of patients to identify the probability of pulmonary hypertension, especially in the early stages of pulmonary hypertension. However, due to the lack of specificity in the early clinical manifestations of pulmonary hypertension, current methods for classifying pulmonary artery images for abnormalities struggle to accurately identify the pathological characteristics of pulmonary hypertension, thus reducing the accuracy of pulmonary artery image anomaly classification. Summary of the Invention

[0004] The main objective of this disclosure is to provide a method, apparatus, device, and medium for classifying abnormalities in pulmonary artery images, which can improve the accuracy of abnormality classification in pulmonary artery images.

[0005] To achieve the above objectives, a first aspect of the present disclosure provides a method for anomaly classification of pulmonary artery images, comprising: acquiring a target CT image and a target ultrasound image of the pulmonary artery;

[0006] The target CT image and the target ultrasound image are encoded separately, and the first global and local features of the target CT image and the second global and local features of the target ultrasound image are extracted after encoding. The first global and local features and the second global and local features are weighted and fused through a preset multi-head attention mechanism to obtain multimodal fusion features.

[0007] The multimodal fusion features are decoded, and the target fusion image of the pulmonary artery is reconstructed after decoding. Anomaly classification is performed based on the target fusion image to obtain the anomaly classification result of the pulmonary artery.

[0008] In some embodiments, the step of encoding the target CT image and the target ultrasound image respectively, and extracting the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding, includes:

[0009] The target CT image and the target ultrasound image are encoded respectively to obtain the first shallow layer features of the target CT image and the second shallow layer features of the target ultrasound image;

[0010] Global feature extraction processing is performed on the first shallow feature and the second shallow feature respectively to obtain the first basic feature corresponding to the target CT image and the second basic feature corresponding to the target ultrasound image;

[0011] Local feature extraction processing is performed on the first shallow feature and the second shallow feature respectively to obtain the first detail feature corresponding to the target CT image and the second detail feature corresponding to the target ultrasound image;

[0012] Based on the first basic feature and the first detailed feature, a first global and local feature of the target CT image is obtained, and based on the second basic feature and the second detailed feature, a second global and local feature of the target ultrasound image is obtained. In some embodiments, encoding the target CT image and the target ultrasound image respectively to obtain a first shallow feature of the target CT image and a second shallow feature of the target ultrasound image includes:

[0013] The target CT image and the target ultrasound image are respectively input into a shared encoder, wherein the shared encoder is provided with a multi-head self-attention layer and a feedforward neural network;

[0014] The target CT image and the target ultrasound image are respectively input into the multi-head self-attention layer, and the first self-attention feature of the target CT image and the second self-attention feature of the target ultrasound image are output.

[0015] The first self-attention feature and the second self-attention feature are respectively input into the feedforward neural network, and the first shallow feature of the target CT image and the second shallow feature of the target ultrasound image are output.

[0016] In some embodiments, obtaining first global and local features of the target CT image based on the first basic features and the first detail features, and obtaining second global and local features of the target ultrasound image based on the second basic features and the second detail features, includes:

[0017] Based on the first basic feature and the first detailed feature, the first initial global and local features of the target CT image are obtained, and based on the second basic feature and the second detailed feature, the second initial global and local features of the target ultrasound image are obtained;

[0018] The first initial global and local features are input into a preset reversible layer, in which the channels of the first initial global and local features are divided into two parts. The latter part of the channel is incrementally transformed based on the mapping function of the former part of the channel, and the former part of the channel is multiplied and biased based on the mapping function of the latter part of the channel. The features of the two parts of the channel are then spliced ​​together to output the first global and local features of the optimized target CT image.

[0019] The second initial global and local features are input into the reversible layer to divide the channels of the second initial global and local features into two parts in the reversible layer. The latter part of the channel is incrementally transformed based on the mapping function of the former part of the channel, and the former part of the channel is multiplied and biased based on the mapping function of the latter part of the channel. The features of the two parts of the channel are then spliced ​​together to output the second global and local features of the optimized target ultrasound image.

[0020] In some embodiments, the step of weightedly fusing the first global and local features and the second global and local features through a preset multi-head attention mechanism to obtain multimodal fused features includes:

[0021] The first global and local features and the second global and local features are input into a preset multi-head attention fusion module;

[0022] In the multi-head attention fusion module, the first global and local features are mapped to a query matrix, and the global and local features are mapped to a key matrix and a value matrix. After calculating the similarity score between the query matrix and the key matrix, the attention weights are normalized to obtain the attention weights. The value matrix is ​​then weighted based on the attention weights to concatenate the outputs of multiple attention heads and obtain multimodal fusion features.

[0023] In some embodiments, decoding the multimodal fusion features and reconstructing the target fusion image of the pulmonary artery after decoding includes:

[0024] The multimodal fusion features are subjected to feature enhancement processing to obtain multimodal enhanced features;

[0025] The multimodal enhancement features are subjected to dimensionality reduction processing to obtain multimodal dimensionality reduction features;

[0026] The multimodal dimensionality reduction features are convolved and activated to map them to the pixel space of the output image, and then normalized to output the target fused image of the pulmonary artery.

[0027] In some embodiments, the abnormal classification based on the target fused image to obtain the abnormal classification result of the pulmonary artery includes:

[0028] The target fused image is subjected to multiple rounds of convolution processing with different time lengths to obtain the first path feature, and the target fused image is subjected to multiple rounds of convolution and pooling processing to obtain the second path feature;

[0029] The first path features and the second path features are merged to obtain multi-scale features.

[0030] The multi-scale features are classified and mapped to output the abnormal classification results of the pulmonary artery.

[0031] To achieve the above objectives, a second aspect of the present disclosure provides an anomaly classification device for pulmonary artery images, comprising: an image acquisition module for acquiring a target CT image and a target ultrasound image of the pulmonary artery;

[0032] The encoding module is used to encode the target CT image and the target ultrasound image respectively, and extract the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding.

[0033] The fusion module is used to perform weighted fusion of the first global and local features and the second global and local features through a preset multi-head attention mechanism to obtain multimodal fused features;

[0034] The decoding module is used to decode the multimodal fusion features and reconstruct the target fusion image of the pulmonary artery after decoding.

[0035] The classification module is used to perform anomaly classification based on the target fused image to obtain the anomaly classification result of the pulmonary artery.

[0036] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the abnormal classification method for pulmonary artery images described in the first aspect embodiment.

[0037] To achieve the above objectives, a fourth aspect of the present disclosure provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the abnormal classification method for pulmonary artery images described in the first aspect of the present disclosure.

[0038] This embodiment of the disclosure obtains a target CT image and a target ultrasound image of the pulmonary artery by performing an anomaly classification method for pulmonary artery images. The target CT image and the target ultrasound image are encoded, and a first global and local feature of the target CT image and a second global and local feature of the target ultrasound image are extracted after encoding. The first and second global and local features are weighted and fused using a preset multi-head attention mechanism to obtain a multimodal fusion feature. The multimodal fusion feature is decoded, and the target fused image of the pulmonary artery is reconstructed after decoding. Anomaly classification is performed based on the target fused image to obtain the anomaly classification result of the pulmonary artery.

[0039] Therefore, this embodiment of the present disclosure acquires two modalities of images—target CT images and target ultrasound images—and leverages their complementarity to capture potential early symptom features from different dimensions, overcoming the limitations of insufficient information from a single image. Subsequently, the two types of images are encoded separately to extract their global and local features, ensuring that subtle pathological changes are not overlooked while preserving overall structural information. Then, a pre-defined multi-head attention mechanism is used to weightedly fuse the aforementioned multimodal features. This mechanism adaptively focuses on the feature subspace most relevant to pulmonary hypertension, strengthening the expression of key symptom signals and suppressing irrelevant interference, thereby generating more discriminative multimodal fusion features. Next, the multimodal fusion features are decoded and the image is reconstructed to obtain the target fused image, transforming the fused abstract features into an intuitive and complete image representation. Finally, classification is performed based on the target fused image, enabling the model to more accurately identify the early specific manifestations of pulmonary hypertension, thus overcoming the classification limitations caused by inaccurate symptom identification in related technologies and ultimately improving the accuracy of pulmonary artery image abnormality classification. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an application environment for the pulmonary artery image anomaly classification method provided in this embodiment of the present disclosure.

[0041] Figure 2 This is a schematic flowchart of the abnormal classification method for pulmonary artery images provided in the embodiments of this disclosure;

[0042] Figure 3 yes Figure 2 A flowchart further includes step S102;

[0043] Figure 4 This is a schematic diagram of the multimodal image fusion model architecture provided in the embodiments of this disclosure;

[0044] Figure 5 yes Figure 3A flowchart further included in step S201;

[0045] Figure 6 yes Figure 3 A flowchart further included in step S204;

[0046] Figure 7 yes Figure 2 A flowchart further includes step S103;

[0047] Figure 8 yes Figure 2 A flowchart further includes step S104;

[0048] Figure 9 yes Figure 2 A flowchart further includes step S105;

[0049] Figure 10 This is a schematic diagram of the pulmonary hypertension classification model architecture provided in the embodiments of this disclosure;

[0050] Figure 11 This is a schematic diagram of CT and ultrasound images of pulmonary hypertension patients and normal patients provided in the embodiments of this disclosure;

[0051] Figure 12 This is a schematic diagram illustrating the influence of different modules on the change of loss value during the CT and ultrasound multimodal fusion process provided in this embodiment of the disclosure;

[0052] Figure 13 This is a schematic diagram illustrating the use of a confusion matrix provided in this embodiment to evaluate the performance of a multimodal pulmonary hypertension image classification model;

[0053] Figure 14 This is a schematic diagram of the functional modules of the pulmonary artery image abnormality classification device provided in this embodiment of the present disclosure;

[0054] Figure 15 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0055] To enable those skilled in the art to better understand the solutions disclosed herein, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0056] It is understood that in the specific embodiments of this disclosure, which involve retrieving initial time-series data, initial sample time-series data and related data, when the above embodiments of this disclosure are applied to specific products or technologies, permission or consent from the target is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0057] Furthermore, when the embodiments of this disclosure require access to initial timing data, initial sample timing data, and related data, separate permission or consent to the initial timing data, initial sample timing data, and related data will be obtained through pop-up windows or redirection to a confirmation page. After clearly obtaining separate permission or consent to the initial timing data, initial sample timing data, and related data, the necessary initial timing data, initial sample timing data, and related data for enabling the embodiments of this disclosure to operate normally will be obtained.

[0058] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0059] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0062] CT (Computed Tomography) is a medical imaging technique that uses X-ray beams to perform layered scans of the human body and then uses computer processing to produce detailed images of the body's internal structures. CT images are medical images generated through computed tomography, displaying the internal structures of the human body layer by layer. They have high resolution and are commonly used to diagnose fractures, acute bleeding, and lung scans.

[0063] Ultrasound imaging is a medical imaging technology based on the propagation characteristics of high-frequency ultrasound waves in human tissues. This technology utilizes the reflected waves generated by the difference in acoustic impedance between tissues to generate grayscale or color Doppler images through signal processing algorithms.

[0064] Convolutional Neural Networks (CNNs) are a type of feedforward neural network widely used in image recognition and vision tasks, and are one of the core models in deep learning. Through a hierarchical structure of convolutional layers, pooling layers, and fully connected layers, they automatically extract local features from input data, progressively compress information, reduce redundancy, and improve generalization ability.

[0065] The pulmonary artery is a blood vessel connecting the right ventricle of the heart to the lungs, responsible for transporting hypoxic venous blood from the heart to the lungs for gas exchange. Pulmonary hypertension is a complex and serious cardiopulmonary vascular disease characterized by elevated pulmonary artery pressure and increased pulmonary vascular resistance, ultimately leading to right heart failure and even death. Because pulmonary hypertension often presents with no obvious symptoms in its early stages, it is easily overlooked. By the time obvious symptoms appear and patients seek medical attention, it is often already in an advanced stage. Therefore, early detection of pulmonary hypertension is crucial for improving patient prognosis. However, due to the lack of specific clinical manifestations of pulmonary hypertension, early detection remains a significant challenge.

[0066] With the development of artificial intelligence technology, neural networks can now be used to classify pulmonary artery images of patients to identify the probability of pulmonary hypertension, especially in the early stages of pulmonary hypertension. However, due to the lack of specificity in the early clinical manifestations of pulmonary hypertension, current methods for classifying pulmonary artery images for abnormalities struggle to accurately identify the pathological characteristics of pulmonary hypertension, thus reducing the accuracy of pulmonary artery image anomaly classification.

[0067] To address the aforementioned problems, this disclosure provides a method, apparatus, device, and medium for classifying abnormalities in pulmonary artery images, which can improve the accuracy of abnormality classification in pulmonary artery images.

[0068] Please see Figure 1 , Figure 1 A schematic diagram of the implementation environment for the abnormal classification method of pulmonary artery images provided in this embodiment of the disclosure includes: a terminal 11 and a server 12.

[0069] For example, server 12 can acquire target CT and target ultrasound images of the pulmonary artery from terminal 11; encode the target CT and target ultrasound images respectively, and extract the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding; use a preset multi-head attention mechanism to perform weighted fusion of the first and second global and local features to obtain multimodal fusion features; decode the multimodal fusion features and reconstruct the target fused image of the pulmonary artery after decoding; perform anomaly classification based on the target fused image to obtain the anomaly classification result of the pulmonary artery. Finally, server 12 can send the final anomaly classification result to terminal 11.

[0070] Terminal 11 can be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, vehicle terminal, etc., but is not limited to these. Terminal 11 can also independently execute the abnormal classification method of pulmonary artery image. Terminal 11 and server 12 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the present disclosure does not impose any limitations.

[0071] Server 12 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 12 can also be a node server in a blockchain network.

[0072] It should be noted that, Figure 1The schematic diagram of the implementation environment shown is merely an example. The scenarios described in this disclosure are intended to more clearly illustrate the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in this disclosure are also applicable to similar technical problems.

[0073] Please see Figure 2 , Figure 2 This is a flowchart illustrating the pulmonary artery image anomaly classification method provided in this embodiment. This pulmonary artery image anomaly classification method can be applied to the server in the above embodiment, or can be jointly executed by a terminal and a server. The pulmonary artery image anomaly classification method includes steps S101 to S105:

[0074] Step S101: Acquire the target CT image and target ultrasound image of the pulmonary artery;

[0075] Step S102: Encode the target CT image and the target ultrasound image respectively, and extract the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding.

[0076] Step S103: Through a preset multi-head attention mechanism, the first global and local features and the second global and local features are weighted and fused to obtain multimodal fused features;

[0077] Step S104: Decode the multimodal fusion features and reconstruct the target fusion image of the pulmonary artery after decoding.

[0078] Step S105: Perform anomaly classification based on the target fusion image to obtain the anomaly classification result of the pulmonary artery.

[0079] Regarding step S101 above, this embodiment of the disclosure can acquire target CT images and target ultrasound images of the pulmonary artery. These target CT images and target ultrasound images belong to patients requiring pulmonary artery image abnormality classification, or are CT images and ultrasound images taken of a specific patient in a sample database. By acquiring CT images and ultrasound images taken of a specific patient, as the data basis for abnormality classification, they can subsequently be used to identify whether pulmonary hypertension exists in the images. It should be noted that, under the premise of meeting the requirements of this embodiment of the disclosure, in addition to image recognition of pulmonary hypertension among abnormalities, targeted training can also be conducted to enable the identification of other lung abnormalities based on CT images and ultrasound images after training. This embodiment of the disclosure does not specifically limit this aspect.

[0080] Among them, the target CT image is a lung CT image taken of the lungs of the target patient. Specifically, it can be a computed tomography image of the pulmonary artery region, which, after standardized preprocessing, is used to extract anatomical structural features. It is one of the core modalities of multimodal data. The target ultrasound image is a lung ultrasound image taken of the lungs of the target patient. Specifically, it can be an echocardiogram image of the heart and pulmonary artery hemodynamics, which, after grayscale and contrast enhancement processing, is used to extract functionally relevant features. It is another core modality of multimodal data.

[0081] Furthermore, embodiments of this disclosure can perform image preprocessing operations after acquiring CT and ultrasound images. For example, preprocessing operations can eliminate image noise, unify data formats, and avoid feature extraction deviations caused by differences in image quality or spatial misalignment. Specifically, if bone tissue is not removed from the CT image, it will interfere with the feature recognition of subtle thickenings in the pulmonary artery wall; if registration is not performed, the vascular morphology features of CT and the blood flow features of ultrasound cannot be accurately correlated. Finally, after preprocessing, target CT and target ultrasound images are obtained for subsequent processes, and embodiments of this disclosure do not impose specific limitations on this.

[0082] It should be noted that CT images are good at showing the morphology and structure of the lungs and heart, but lack specificity for the diagnosis of early pulmonary hypertension. Ultrasound images can estimate pulmonary artery pressure, but the accuracy is affected by the operator's experience and the patient's body size, and the sensitivity is low for mild pulmonary hypertension. The two modalities of images are naturally complementary, laying the foundation for subsequent feature extraction.

[0083] Regarding step S102 above, this embodiment of the present disclosure can encode the preprocessed target CT image and target ultrasound image respectively, and extract corresponding global and local features from them. This transforms the intuitive image data into computer-analyzable abstract feature vectors. Through dual-dimensional feature extraction of both global and local features, it ensures that no overall structural abnormalities or subtle pathological changes that may exist in the early stages of pulmonary hypertension are missed. This allows for the capture of overall image correlation information while also uncovering key local symptom signals, providing high-quality data support for subsequent multimodal feature fusion. It should be noted that, provided the feature extraction accuracy requirements of this embodiment of the present disclosure are met, this encoding logic is not only applicable to pulmonary artery images, but can also be extended to image feature extraction of other lung diseases such as pulmonary nodules and pulmonary embolism by adjusting encoder parameters (such as convolution kernel size and number of attention heads). This embodiment of the present disclosure does not impose specific limitations on this.

[0084] The encoding process refers to the process of mapping target CT and ultrasound images from pixel space to a high-dimensional feature space using a pre-defined dual-branch encoder architecture. The core is to leverage the synergy between Transformer and CNN to extract structural, textural, and semantic information from the images, discarding redundant data in the original pixels, such as device noise and irrelevant tissue signals. For example, the encoder can first extract shallow features using Transformer blocks, and then enhance global features and extract local features.

[0085] The first global and local features are feature sets extracted from the target CT image. They can include the first global features and the first local features. The first global features focus on the overall structural relationship between the pulmonary artery and the lung, such as the diameter of the main pulmonary artery, the density of pulmonary vascular branches, and the spatial layout of the right ventricle and the pulmonary artery, which can reflect the pathological trend at the overall level of the image. The first local features focus on the subtle structural changes of the pulmonary artery, such as local thickening of the vessel wall, traces of stenosis of branch vessels, and the relationship between small lesions in the lung, which can capture hidden lesions at the local level of the image. Together, they constitute the core feature expression of the CT modality.

[0086] The second set of global and local features is a feature set extracted from the target ultrasound image. It may include the second global feature and the second local feature. The second global feature focuses on the overall functional relationship between the heart and the pulmonary artery, such as the right ventricular end-diastolic volume, the overall blood flow direction of the pulmonary artery, and the coordinated movement of the heart valves and the pulmonary artery, which can reflect the overall abnormality at the functional level. The second local feature focuses on subtle changes in hemodynamics, such as the signal intensity of tricuspid regurgitation, fluctuations in regurgitation velocity, and subtle defects at the valve edge, which can capture hidden abnormalities at the functional level. Together, the two constitute the core feature expression of the ultrasound modality.

[0087] Furthermore, the encoding processing in this embodiment must follow the modality adaptation principle. For the target CT image, the attention window size can be appropriately increased to enhance overall structural capture; for the target ultrasound image, the attention window can be reduced to focus on the functionally related regions of the heart and pulmonary arteries, and contrast-sensitive convolution can be combined to highlight local fluctuations in blood flow signals. If the encoding processing does not distinguish modal characteristics, it may lead to the loss of local details of the blood flow signal; if only a single encoder is relied upon, abnormal trends in the overall structure of the pulmonary artery will be missed, ultimately affecting the subsequent fusion effect. Therefore, this embodiment can adjust the encoder parameters according to the actual image modality without specific limitations.

[0088] It should be noted that the embodiments of this disclosure achieve collaborative extraction of global and local features through a dual-branch encoder. The structural features of the CT modality and the functional features of the ultrasound modality are encoded to form a complementary feature set, which lays the foundation for the weighted fusion of the subsequent multi-head attention mechanism and avoids the symptom recognition bias caused by insufficient feature information of a single modality.

[0089] Regarding step S103 above, this embodiment of the present disclosure can use a preset multi-head attention mechanism to perform weighted fusion of the first global and local features and the second global and local features to obtain multimodal fusion features. Furthermore, through adaptive weight allocation, it strengthens key feature signals related to pulmonary hypertension, such as the associated features of CT vessel wall thickening and increased ultrasound reflux velocity, while suppressing irrelevant redundant information, such as normal lung texture and ultrasound equipment noise. This achieves synergistic enhancement of dual-modal features, providing a more discriminative feature foundation for subsequent decoding, reconstruction, and classification.

[0090] The pre-defined multi-head attention mechanism is a feature fusion technique based on the Transformer architecture. Its core is to mine feature dependencies across different dimensions using multiple parallel attention heads and adaptively assign feature weights. Specifically, this mechanism first linearly transforms the features into query (Q), key (K), and value (V) matrices. Then, each attention head independently calculates the similarity between the query and key matrices to obtain corresponding weights. Finally, the outputs of multiple attention heads are concatenated and linearly transformed to obtain a unified fused feature. The multimodal fusion feature refers to a high-dimensional feature vector formed by weighted fusion through the multi-head attention mechanism. It integrates the structural features of the target CT image and the functional features of the target ultrasound image, and highlights the specific feature combination of pulmonary hypertension through weight allocation. This feature avoids the information limitations of single-modal features and eliminates the redundant interference of dual-modal features.

[0091] Furthermore, this embodiment of the present disclosure can introduce LayerNorm (layer normalization) and DropPath (random path discarding) techniques after weighted fusion. LayerNorm reduces internal covariate bias by normalizing the feature mean and variance, accelerating the convergence speed of feature fusion. DropPath increases the model regularization effect by randomly discarding 5%-10% of the feature paths, avoiding overfitting during the fusion process. The two techniques work synergistically to further improve the stability and generalization ability of multimodal fused features.

[0092] It should be noted that the multi-head attention mechanism in this embodiment can dynamically capture feature associations, avoiding feature redundancy caused by direct splicing, thereby enabling adaptive focus on key features and improving the discriminative power of fused features.

[0093] Regarding step S104 above, the embodiments of this disclosure can decode the multimodal fusion features and reconstruct the target fusion image of the pulmonary artery after decoding, thereby transforming the abstract multimodal fusion features into an intuitive visual image, providing interpretable and easily discriminated image input for subsequent abnormal classification, and facilitating clinicians to intuitively verify the fusion effect.

[0094] Decoding is the inverse process of encoding, restoring multimodal fusion features from a high-dimensional feature space to a pixel space. The target fusion image is a visual image obtained through decoding and reconstruction, integrating the structural representation of the target CT image and the functional representation of the target ultrasound image. This image overcomes the limitations of missing functional information in CT images and blurred structural details in ultrasound images, while providing intuitive input for subsequent classification that combines structure and function.

[0095] It should be noted that the embodiments of this disclosure obtain the target fusion image through decoding and reconstruction, which not only solves the feature interpretability problem, but also provides clinicians with intuitive auxiliary diagnostic basis, effectively improving the accuracy and reliability of subsequent classification.

[0096] Regarding step S105 above, the embodiments of this disclosure can perform anomaly classification based on the reconstructed target fusion image to finally obtain the abnormal classification result of the pulmonary artery. For example, a preset CNN classification model can be used to automatically identify the specific characteristics of pulmonary hypertension in the target fusion image and output a normal or abnormal conclusion, thus solving the classification error problem caused by insufficient single modality features and inaccurate symptom identification.

[0097] Among them, the abnormal classification result refers to the conclusion of the pulmonary artery health status output after the target fused image is analyzed by the classification model. For example, if the output abnormal classification result is 1, it means that the image is abnormal, that is, there may be pulmonary hypertension. Conversely, if the output abnormal classification result is 0, it means that the image is normal, that is, there is no pulmonary hypertension.

[0098] It should be noted that, compared to single-modal image classification (such as CT or ultrasound only), which suffers from high false negative rates and low accuracy, this embodiment of the present disclosure is based on target fusion image classification. It integrates complementary information from both modalities and accurately captures synergistic abnormalities in structure and function, effectively overcoming previous classification limitations and significantly improving the accuracy of abnormal classification in pulmonary artery images. In summary, this embodiment of the present disclosure, by executing the abnormal classification method for pulmonary artery images in steps S101 to S105, acquires images from two modalities—target CT images and target ultrasound images—and utilizes their complementarity to capture potential early symptom features from different dimensions, overcoming the limitations of insufficient information from a single image. Subsequently, the two types of images are encoded separately to extract their global and local features, ensuring that subtle pathological changes are not missed while preserving overall structural information. Then, a pre-set multi-head attention mechanism is used to weightedly fuse the above-mentioned multimodal features. This mechanism adaptively focuses on the feature subspace most relevant to pulmonary hypertension, strengthens the expression of key symptom signals, and suppresses irrelevant interference, thereby generating multimodal fusion features with stronger discriminative power. Next, by decoding and reconstructing the multimodal fusion features, the target fused image is obtained, transforming the abstract features into an intuitive and complete image representation. Finally, classification is performed based on the target fused image, enabling the model to more accurately identify the early specific manifestations of pulmonary hypertension. This overcomes the classification limitations caused by inaccurate symptom identification in related technologies, ultimately improving the accuracy of pulmonary artery image abnormality classification.

[0099] The following is a detailed description of the further contents included in steps S101 to S105 in the embodiments of this disclosure.

[0100] Please see Figure 3 , Figure 3 yes Figure 2 The flowchart further includes step S102. In some embodiments, the process of encoding the target CT image and the target ultrasound image respectively, and extracting the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding, may include steps S201 to S204:

[0101] Step S201: Encode the target CT image and the target ultrasound image respectively to obtain the first shallow layer features of the target CT image and the second shallow layer features of the target ultrasound image.

[0102] Step S202: Perform global feature extraction processing on the first shallow layer features and the second shallow layer features respectively to obtain the first basic features corresponding to the target CT image and the second basic features corresponding to the target ultrasound image;

[0103] Step S203: Local feature extraction processing is performed on the first shallow layer feature and the second shallow layer feature respectively to obtain the first detail feature corresponding to the target CT image and the second detail feature corresponding to the target ultrasound image;

[0104] Step S204: Obtain the first global and local features of the target CT image based on the first basic features and the first detailed features, and obtain the second global and local features of the target ultrasound image based on the second basic features and the second detailed features.

[0105] Please see Figure 4 , Figure 4 This is a schematic diagram of the multimodal image fusion model architecture provided in this embodiment. In this embodiment, the multimodal image fusion model can receive input target CT images (i.e., the CT image in the figure) and target ultrasound images (i.e., the ultrasound image in the figure), and after processing by various internal layers, finally output the target fused image (i.e., the fused image in the figure). The multimodal image fusion model includes an encoder, comprising a shared encoder (i.e., the shared coding layer - Restor module in the figure), a base layer (base transform encoder (BTE)), detail layers, an invertible neural network (INN) module, a multi-head attention module, and a decoder (i.e., the decoding layer - Restor module in the figure), etc. The following describes the architecture in conjunction with... Figure 4 The embodiments of this disclosure will be described.

[0106] In the above steps, the present invention can encode the target CT image and the target ultrasound image respectively, and output the first shallow layer features of the target CT image and the second shallow layer features of the target ultrasound image, thereby transforming the pixel-level raw image data into low-dimensional abstract features that can be preliminarily analyzed by a computer, preserving the basic structure and semantic association of the image, and providing feature support for subsequent global feature extraction and local feature extraction.

[0107] The first shallow feature is the low-level abstract feature obtained after encoding the target CT image. It is generated by capturing feature dependencies through a multi-head self-attention layer of a shared encoder and enhancing feature representation through a feedforward neural network. The second shallow feature is the low-level abstract feature obtained after encoding the target ultrasound image. It is also generated through a multi-head self-attention layer of a shared encoder and a feedforward neural network.

[0108] Next, in this embodiment, global feature extraction processing can be performed on the first and second superficial features respectively, outputting the first basic feature corresponding to the target CT image and the second basic feature corresponding to the target ultrasound image. This is to mine the overall structural and functional correlation information from the underlying superficial features, capture the possible global pathological trends in the early stage of pulmonary hypertension, and avoid focusing only on local features while ignoring the overall symptom correlation, thus providing a macroscopic feature dimension for the final integration of global and local features. The first basic feature is a macroscopic feature obtained after global extraction of the first superficial features of the target CT image. Its core includes the overall structural correlation information of the pulmonary artery and lungs. Furthermore, the first basic feature can be generated by the basis transformer encoder (BTE) in the encoder. The BTE focuses on the globally correlated region of the image through an attention mechanism, and combines an MLP network to enhance the global information of the superficial features, ultimately outputting the first basic feature with the same dimension as the first superficial feature. Each dimension of its feature vector corresponds to the quantitative expression of a certain global structure in the CT image. The second basic feature is a macroscopic feature obtained by globally extracting the second shallow feature of the target ultrasound image. Similar to the first basic feature, it is also generated by the basis transformer encoder (BTE). The BTE adjusts the focusing range of the attention mechanism according to the functional characteristics of the ultrasound modality, strengthens the global association at the functional level through the MLP network, and outputs the second basic feature with the same dimension as the second shallow feature. Each dimension of its feature vector corresponds to the quantitative expression of a certain global function of the ultrasound image.

[0109] It should be noted that the embodiments disclosed herein extract global basic features through BTE, which can accurately capture macro trends and provide an overall pathological background for subsequent integration of local features.

[0110] Subsequently, the embodiments of this disclosure can perform local feature extraction processing on the first and second superficial features respectively, and output the first detail feature corresponding to the target CT image and the second detail feature corresponding to the target ultrasound image, so as to mine local subtle pathological change information from the bottom superficial features and capture the local hidden symptoms that may exist in the early stage of pulmonary hypertension. These subtle changes are often specific markers of early PH, but they are easily masked by global features. Therefore, they need to be enhanced by a dedicated local extraction module to provide microscopic symptom support for the final integration of global and local features.

[0111] The first detail feature is a microscopic feature obtained by locally extracting the first shallow features of the target CT image. It is generated by the detail layer, i.e., the detail CNN encoder (DCE). The DCE refines the local regions in the shallow features through inverse residual blocks and detail nodes, and finally outputs the first detail feature with the same dimension as the first shallow feature. Each dimension of its feature vector corresponds to a subtle quantitative difference in a local structure of the CT image. The second detail feature is a microscopic feature obtained by locally extracting the second shallow features of the target ultrasound image. It is also generated by the detail CNN encoder (DCE). The DCE adjusts the number of channels in the inverse residual block and the receptive field of the detail nodes according to the functional characteristics of the ultrasound modality, specifically capturing the local fluctuations of the functional signal, and outputs the second detail feature with the same dimension as the second shallow feature. Each dimension of its feature vector corresponds to a subtle quantitative difference in a local function of the ultrasound image.

[0112] Furthermore, the local feature extraction in this embodiment of the present disclosure needs to follow the high-frequency focusing principle: For the local structural details of the target CT image, DCE uses a 3×3 small convolution kernel, which can accurately capture high-frequency structural information such as vessel wall thickness and lumen stenosis, and sets 2-3 inverse residual blocks to avoid the loss of details caused by excessive convolution; For the local functional details of the target ultrasound image, DCE adds a 1×1 convolution kernel on the basis of the 3×3 convolution kernel to enhance high-frequency functional signals such as reflux signal and blood flow velocity, and sets 3-4 inverse residual blocks. The ultrasound signal noise is stronger, and more modules are needed to filter noise and retain details. In this regard, the DCE parameters can be adjusted according to the local feature characteristics of the modality, without specific limitations.

[0113] It should be noted that the embodiments of this disclosure extract local detailed features through DCE, which can accurately capture hidden symptoms and provide key microscopic information for subsequent feature integration.

[0114] Finally, in this embodiment, firstly, based on the first basic feature and the first detailed feature, first global and local features of the target CT image are generated; simultaneously, based on the second basic feature and the second detailed feature, second global and local features of the target ultrasound image are generated. It should be noted that, provided that the feature integration accuracy and information retention rate are satisfied, the obtained global and local features can be optimized through a reversible layer. The optimization logic can be extended to other multi-feature fusion scenarios, and this embodiment does not impose specific limitations on this.

[0115] Please see Figure 5 , Figure 5 yes Figure 3The flowchart further includes step S201. In some embodiments, the process of encoding the target CT image and the target ultrasound image respectively to obtain the first superficial feature of the target CT image and the second superficial feature of the target ultrasound image may include steps S301 to S303:

[0116] Step S301: Input the target CT image and the target ultrasound image into the shared encoder, wherein the shared encoder is configured with a multi-head self-attention layer and a feedforward neural network;

[0117] Step S302: Input the target CT image and the target ultrasound image into the multi-head self-attention layer respectively, and output the first self-attention feature of the target CT image and the second self-attention feature of the target ultrasound image;

[0118] Step S303: Input the first self-attention feature and the second self-attention feature into the feedforward neural network respectively, and output the first shallow layer feature of the target CT image and the second shallow layer feature of the target ultrasound image.

[0119] In the above steps, the present invention can input the target CT image and the target ultrasound image into a preset shared encoder respectively, providing a unified underlying encoding logic for the two modal images, avoiding incompatibility of encoding rules due to the difference in imaging principles between CT and ultrasound, ensuring that the features extracted subsequently have consistency in dimensionality and semantic expression, and laying the foundation for cross-modal feature association and fusion.

[0120] The shared encoder refers to an integrated encoding module that simultaneously adapts to both CT and ultrasound modalities, including a multi-head self-attention layer and a feedforward neural network. Through unified parameter initialization and activation functions, it achieves consistent yet adaptable encoding for different modalities, ensuring both consistent encoding logic and adaptive capture of core information based on the modal's inherent data characteristics. Next, in this embodiment, the target CT image and target ultrasound image input to the shared encoder can be respectively passed to the encoder's multi-head self-attention layer, ultimately outputting the first self-attention feature of the target CT image and the second self-attention feature of the target ultrasound image.

[0121] The multi-head self-attention layer is a feature processing module based on the Transformer architecture. It linearly transforms the input image features into a query, key, and value matrix, and then calculates the corresponding self-attention features. The first self-attention feature is the feature obtained after processing the target CT image by the multi-head self-attention layer, and the second self-attention feature is the feature obtained after processing the target ultrasound image by the multi-head self-attention layer.

[0122] It should be noted that traditional encoding in related technologies often uses local convolution of CNNs, which can only capture local correlations within a 3×3 or 5×5 range and cannot see the global dependency between the main pulmonary artery and its distal branches, resulting in the neglect of the features of the overall structural shift in the early stage of PH. However, the multi-head self-attention layer of the present disclosure embodiment can realize dependency modeling across the entire image, which can effectively make up for the limitations of traditional CNNs and provide more comprehensive structured information for subsequent shallow feature generation.

[0123] Next, in this embodiment of the present disclosure, the first self-attention feature and the second self-attention feature can be respectively input into the feedforward neural network of the shared encoder, and finally output the first shallow feature of the target CT image and the second shallow feature of the target ultrasound image, thereby enhancing the expressive power of the self-attention feature, eliminating noise interference, and regularizing the feature dimension. The feedforward neural network can enhance the effective signal and filter the invalid noise through nonlinear transformation, and regularize the feature dimension to a uniform size.

[0124] The feedforward neural network is the component in the shared encoder used for feature optimization. It employs a three-stage architecture: linear transformation, ReLU activation, and another linear transformation. The first stage, linear transformation, maps the self-attention features to a higher dimension, providing a richer expression space for nonlinear transformations. The ReLU activation function enhances the effective signal by suppressing negative eigenvalues. The second stage, linear transformation, maps the high-dimensional features back to the target dimension, achieving feature dimension regularization and further strengthening the core signal. The first shallow feature is the low-level abstract feature obtained by optimizing the first self-attention feature of the target CT image using the feedforward neural network. The second shallow feature is the low-level abstract feature obtained by optimizing the second self-attention feature of the target ultrasound image using the feedforward neural network.

[0125] Furthermore, the output of the shared encoder is as follows:

[0126]

[0127] Where S() represents the output of the multi-head self-attention machine layer and the feedforward neural network. I represents the representation information of the target CT image, and V represents the representation information of the target ultrasound image. and These represent the first and second shallow features extracted, respectively. Furthermore, the encoder incorporates a Basis Transformer Encoder (BTE) and a Details CNN Encoder (DCE), used to extract basic and detailed features based on the shallow features, respectively. The Basis Transformer Encoder enhances and fuses the features through an attention mechanism and an MLP network. The resulting first and second basic features are as follows:

[0128]

[0129] Where B() represents the output of BTE. I represents the representation information of the target CT image, and V represents the representation information of the target ultrasound image. and These represent the first and second basic features extracted, respectively.

[0130] DCE further extracts image detail information through inverse residual blocks and detail nodes. This dual-path feature extraction strategy enables the model to simultaneously capture both global features and local details of the image, providing rich feature representations for subsequent fusion and classification tasks. The obtained first and second detail features are as follows:

[0131]

[0132] Where D() represents the output of DCE. I represents the representation information of the target CT image, and V represents the representation information of the target ultrasound image. and These represent the extracted first detail feature and the second detail feature, respectively.

[0133] Please see Figure 6 , Figure 6 yes Figure 3 The flowchart further includes step S204. In some embodiments, the process of obtaining first global and local features of the target CT image based on first basic features and first detailed features, and obtaining second global and local features of the target ultrasound image based on second basic features and second detailed features, may include steps S401 to S403:

[0134] Step S401: Obtain the first initial global and local features of the target CT image based on the first basic features and the first detailed features, and obtain the second initial global and local features of the target ultrasound image based on the second basic features and the second detailed features;

[0135] Step S402: Input the first initial global and local features into a preset reversible layer to divide the channels of the first initial global and local features into two parts in the reversible layer. The latter part of the channel is incrementally transformed based on the mapping function of the former part of the channel, and the former part of the channel is multiplied and biased based on the mapping function of the latter part of the channel. The features of the two parts of the channel are then stitched together to output the first global and local features of the optimized target CT image.

[0136] Step S403: Input the second initial global and local features into the reversible layer to divide the channels of the second initial global and local features into two parts in the reversible layer. The latter part of the channel is incrementally transformed based on the mapping function of the former part of the channel, and the former part of the channel is multiplied and biased based on the mapping function of the latter part of the channel. The features of the two parts of the channel are then spliced ​​together to output the second global and local features of the optimized target ultrasound image.

[0137] In the above steps, embodiments of this disclosure can integrate and generate first initial global and local features of the target CT image based on first basic features and first detailed features of the target CT image; and simultaneously integrate and generate second initial global and local features of the target ultrasound image based on second basic features and second detailed features of the target ultrasound image.

[0138] The first initial global and local features are the preliminary fusion results of the first basic features and the first detailed features of the target CT image. The second initial global and local features are the preliminary fusion results of the second basic features and the second detailed features of the target ultrasound image.

[0139] Next, in this embodiment, the first initial global and local features can be input into a preset reversible layer. Through channel segmentation, bidirectional transformation and feature stitching, the first global and local features of the optimized target CT image are output. This achieves deep collaboration of global and local features without information loss, solves the problems of redundancy and loose correlation of global and local information in the initial features, strengthens the global and local correlation signals of pulmonary hypertension-related structures through reversible transformation, and ensures that feature information is not lost. This provides highly discriminative and non-redundant structural features for subsequent multimodal fusion.

[0140] The reversible layer is the core optimization module designed based on reversible feature transformation. Composed of INN modules, its core principle is to achieve lossless information processing through bidirectional mapping. It divides the feature channel into two parts: the first part focuses on global information, and the second part focuses on local information. A bidirectional correlation is established through a mapping function. Local information is adjusted based on global information, and global information is corrected based on local information. Finally, the optimized features are stitched together, and the entire process is reversible, ensuring no information loss. The first global and local features of the optimized target CT image are the final structural features obtained after processing the initial global and local features by the reversible layer.

[0141] Furthermore, this embodiment of the present disclosure uses the example of inputting first initial global and local features into a preset reversible layer and outputting optimized first global and local features of the target CT image to illustrate the processing procedure in the reversible layer. After extracting the aforementioned first initial global and local features, feature extraction and transformation are performed on the reversible layer. The purpose is to further ensure the reversibility between the input and output features, so that information is not lost during transmission, which is beneficial for extracting high-frequency features. The calculation is as follows:

[0142]

[0143] in, This represents the characteristics of the (c+1)th to (c+C)th channels in the (k+1)th reversible layer. Ψ1 represents the feature of the (c+1)th to Cth channels in the k-th reversible layer, and Ψ1 is an arbitrary mapping function used to generate the increment to be added.

[0144] This represents the characteristics of the first to c channels in the (k+1)th reversible layer. This represents the features of channels 1 to c in the k-th invertible layer. ⊙ represents the Hadamard product (element-wise multiplication), exp represents the exponential function, Ψ2 is an arbitrary mapping function used to generate the factors to be multiplied, and Ψ3 is an arbitrary mapping function used to generate the bias to be added. CAT represents the channel concatenation operation, used to concatenate the features of two parts to form the complete output features. It is the output feature of the (k+1)th reversible layer, which contains the information after detransformation, that is, the first global and local features of the final optimized target CT image.

[0145] Similarly, in this embodiment, the second initial global and local features can be input into a preset reversible layer, and the optimized target ultrasound image's second global and local features can be output using the same process as step S402, including channel segmentation, bidirectional transformation, and feature stitching. Further details are omitted here. The optimized target ultrasound image's second global and local features are the final functional features resulting from the processing of the second initial global and local features by the reversible layer.

[0146] Furthermore, although the reversible layer processing in this embodiment is consistent with the process of step S402, it is necessary to adjust the mapping function parameters according to the functional modal characteristics of the ultrasound image to ensure that it is adapted to the fluctuation characteristics of the hemodynamic signal. The adapted reversible layer can retain the functional detail advantages of ultrasound and strengthen the association with the global function. In this regard, this embodiment can adjust the coefficients of Ψ1, Ψ2, and Ψ3 according to the fluctuation amplitude of the ultrasound signal without making specific limitations.

[0147] Please see Figure 7, Figure 7 yes Figure 2 The flowchart further includes step S103. In some embodiments, the process of obtaining multimodal fused features by weighted fusion of the first global and local features and the second global and local features through a preset multi-head attention mechanism may include steps S501 to S502:

[0148] Step S501: Input the first global and local features and the second global and local features into the preset multi-head attention fusion module;

[0149] In step S502, in the multi-head attention fusion module, the first global and local features are mapped to a query matrix, and the second global and local features are mapped to a key matrix and a value matrix. After calculating the similarity scores of the query matrix and the key matrix, the attention weights are normalized. The value matrix is ​​then weighted based on the attention weights to concatenate the outputs of multiple attention heads and obtain multimodal fusion features.

[0150] In the above steps, the embodiments of this disclosure can input the first global and local features of the target CT image (e.g., the features optimized in step S402) and the second global and local features of the target ultrasound image (e.g., the features optimized in step S403) into a preset multi-head attention fusion module.

[0151] The multi-head attention fusion module is a dedicated fusion component designed based on the Transformer multi-head attention mechanism. Its core includes a linear transformation layer, a multi-attention head computation layer, and a feature concatenation layer. The linear transformation layer is responsible for mapping the two input modal features to the same dimension. The multi-attention head computation layer captures the inter-modal dependencies across different dimensions using multiple parallel attention heads. The feature concatenation layer integrates the outputs of multiple attention heads to form a unified multi-modal fusion feature prototype. Furthermore, this embodiment selects a dedicated multi-head attention fusion module, which can unify the feature dimensions through linear transformation at the input stage, clearing obstacles for subsequent dynamic weight allocation and ensuring that the two modal features are fused on the same dimension and according to the same rules. If the input is not processed through this module, subsequent weighted fusion may result in a decrease in the discriminative power of the fused features due to incompatible or loosely correlated feature dimensions. This embodiment does not impose specific limitations on this aspect.

[0152] Next, in the multi-head attention fusion module, the first global and local features are fused with the second global and local features into a multimodal fusion feature through a process of feature mapping, similarity calculation, weight allocation, weighted integration, and multi-head splicing. The core purpose of this step is to achieve adaptive multimodal fusion focusing on key features of pulmonary hypertension. By dynamically calculating the pathological correlation between the two modal features, high weights are assigned to features strongly correlated with pulmonary hypertension, and low weights are assigned to irrelevant features. Ultimately, a fusion feature with strong discriminative power and low redundancy is generated, overcoming the limitation of traditional fixed-weight fusion that cannot adapt to individual pathological differences.

[0153] The query matrix (Q) is generated by linear transformation of the first global and local features. For example, a 1024-dimensional CT feature is transformed into a 512×64 query matrix, where 64 represents the feature sequence length. Its core function is to anchor the structural pathology benchmark. Using the structural abnormalities of the pulmonary artery in the CT image as a reference, it actively seeks matching functional abnormality signals in the ultrasound features. It serves as the reference anchor point for attention weight calculation, ensuring that the fusion process is based on structural abnormalities. The key matrix (K) is generated by linear transformation of the second global and local features, with the same dimensions as the query matrix. Its core function is to provide functional feature matching objects, including all functional signals related to pulmonary hypertension in ultrasound. This allows the query matrix to calculate similarity and filter out the functional signals most relevant to the structural abnormalities, avoiding interference from irrelevant functional signals during fusion. The value matrix (V) is a linearly transformed matrix of the same origin as the key matrix. Its core function is to carry the core information of the functional features. After the attention weights are calculated between the key and query, the value is weighted according to the weights. The value information corresponding to a high-weight key is strengthened, while the value information corresponding to a low-weight key is suppressed, ultimately outputting functional information strongly correlated with structural abnormalities. Attention weights are weights obtained by calculating the dot product similarity between the query matrix and the key matrix, followed by softmax normalization. Their core function is to dynamically filter key collaborative features. Multimodal fusion features are feature matrices formed by concatenating the weighted outputs of multiple attention heads, followed by linear transformation and normalization to obtain the final feature vector. They contain strong collaborative information on CT structural abnormalities and ultrasound functional abnormalities, and their discriminative power is significantly better than that of single-modal features.

[0154] For example, the multi-head attention mechanism in this embodiment is the core of multimodal fusion. Through multi-head self-attention, input features are weighted and summed to highlight important features and suppress unimportant ones. This mechanism not only captures long-distance dependencies between features but also dynamically adjusts feature importance, enabling the model to better adapt to the characteristics of different modalities. The following is the formula for the multi-head attention mechanism, assuming the input sequence is X∈R. n×dWhere n is the sequence length and d is the input dimension. The multi-head attention mechanism first linearly transforms the input into a query, key, and value matrix. Then, it computes attention scores in parallel using multiple attention heads. Finally, the results are concatenated and linearly transformed to obtain the final output: Multihead(X) = concat(h1,h2,h3,h4,...,h...). n W o ;

[0155]

[0156] in, It is the query, key, and value weight matrix of the i-th attention head.

[0157] It is the weight matrix for the output linear transformation. d k `h` represents the dimension of the query and key matrix for each attention head, and `h` is the number of attention heads. `Concat` concatenates the outputs of multiple attention heads. Multi-head attention mechanisms, through parallel computation by multiple attention heads, can capture various dependencies between different positions in the input sequence, thereby improving the model's expressive power and performance.

[0158] Furthermore, embodiments of this disclosure may incorporate techniques such as LayerNorm and DropPath to enhance the training stability and generalization ability of the model. LayerNorm reduces internal covariate shifts by normalizing features, thereby accelerating model convergence; while DropPath increases the regularization effect of the model by randomly discarding parts of the path, preventing overfitting.

[0159] It should be noted that traditional multimodal fusion often uses fixed weights or simple splicing, which cannot adapt to the weak synergistic features in the early stage of pulmonary hypertension, resulting in a high rate of false negatives in classification. However, the embodiments of this disclosure can effectively overcome the limitations of traditional technology through dynamic weighted fusion in this step, accurately capture the structural and functional synergistic features in the early stage of pulmonary hypertension, and provide a highly discriminative feature basis for subsequent decoding, reconstruction and classification.

[0160] Please see Figure 8 , Figure 8 yes Figure 2 The flowchart further includes step S104. In some embodiments, the process of decoding the multimodal fusion features and reconstructing the target fusion image of the pulmonary artery after decoding may include steps S601 to S603:

[0161] Step S601: Perform feature enhancement processing on the multimodal fusion features to obtain multimodal enhanced features;

[0162] Step S602: Perform dimensionality reduction processing on the multimodal enhanced features to obtain multimodal dimensionality-reduced features;

[0163] Step S603: Perform convolution and activation processing on the multimodal dimensionality reduction features to map them to the pixel space of the output image, and then perform normalization processing to output the target fused image of the pulmonary artery.

[0164] In the above steps, embodiments of this disclosure can perform feature enhancement processing on the multimodal fusion features to output multimodal enhanced features. Although the multimodal fusion features have integrated bimodal information, the weak correlation signals in the early stages of pulmonary artery disease may still be masked by redundant information. Feature enhancement can amplify these key pathological correlations while eliminating residual device noise and irrelevant tissue signals during the fusion process.

[0165] Among them, the multimodal enhancement feature is the optimized feature after the multimodal fusion feature has been enhanced. For example, it can be achieved through attention focusing and feature normalization. For instance, the attention focusing module assigns high weights to the pulmonary hypertension-related feature subspace to amplify weak pathological signals, and the feature normalization module reduces internal covariate shift and suppresses noise interference by standardizing the feature mean and variance, and finally outputs an enhanced feature with the same dimension as the multimodal fusion feature and more prominent key signals.

[0166] Next, the embodiments of this disclosure can perform dimensionality reduction processing on the multimodal enhanced features and output multimodal dimensionality-reduced features to reduce the feature dimension and computational complexity while retaining the key pathological information of pulmonary hypertension. Dimensionality reduction can simplify the feature dimension and at the same time screen out the core features with the most diagnostic value.

[0167] Among them, the multimodal dimensionality reduction feature is a low-dimensional feature vector after the multimodal enhancement feature is dimensionality reduced. The core information that is strongly correlated with the diagnosis of pulmonary hypertension in the multimodal enhancement feature is retained, while redundant normal tissue signals and equipment noise signals are removed, and the integrity of the core pathological association is maintained.

[0168] Subsequently, embodiments of this disclosure can perform convolutional mapping, activation enhancement, and pixel normalization on the multimodal dimensionality reduction features, and finally output the target fusion image of the pulmonary artery, thereby transforming the abstract multimodal dimensionality reduction features into an intuitive and complete visualization image.

[0169] The convolutional processing utilizes a decoder-based convolutional layer design, primarily using 3×3 convolutional kernels supplemented by 1×1 kernels. This gradually maps multimodal dimensionality reduction features to the image pixel space. The 3×3 kernels capture spatial relationships between pixels, ensuring the structural integrity of the reconstructed image, while the 1×1 kernels adjust the number of feature channels to suit the dimensionality requirements of the pixel space, avoiding image blurring caused by channel redundancy. Activation processing employs the LeakyReLU activation function to perform a non-linear transformation on the feature values ​​output by the convolution. Normalization uses the Sigmoid function to normalize the activated pixel values ​​to the [0,1] interval, ensuring that the brightness and contrast of the reconstructed image conform to clinical observation practices. The target fusion image is the final output image.

[0170] Based on this, the decoder part of the model is responsible for fusing and reconstructing the features extracted by the encoder to generate the final output. The decoder further processes and enhances the features by constructing a Transformer module. In addition, the decoder also introduces a dimensionality reduction module to reduce the number of feature channels and reduce computational complexity. In the reconstruction stage, the decoder maps the features to the output space through convolutional layers and the LeakyReLU activation function, and normalizes the output values ​​to the range [0,1] using the Sigmoid function. This design not only preserves the detailed information of the input image, but also generates high-quality output results through feature fusion and reconstruction.

[0171] Furthermore, in order to capture local and global information in CT and ultrasound images, this embodiment designs a multi-scale coordinated learning objective function, which specifically includes the following parts:

[0172] Convolution to Representation (CR) Objective: This objective aims to maximize the correlation between local and global information within the same modality. Specifically, feature representations of CT and ultrasound images are extracted through convolutional layers, and then these features are mapped to a shared representation space using fully connected layers. By maximizing the mutual information between feature representations at different scales within the same modality, the model is able to better capture structural information within the modality.

[0173] Cross-convolution to the representation (XX) target: This target is used to capture local-global interaction information between CT and ultrasound images. Features from CT and ultrasound images are fused through cross-convolution operations, and the InfoNCE loss function is used to maximize the mutual information between the two modal feature representations. The InfoNCE loss function encourages the model to learn discriminative and information-rich representations through contrastive learning.

[0174] Representation to Representation (RR) Objective: This objective emphasizes the consistency of global information between the two modalities. By calculating the cosine similarity between the global feature representations of CT and ultrasound images and using mean squared error (MSE) as the loss function, the consistency of global information between the two modalities in the representation space is ensured.

[0175] Convolution-to-Convolution (CC) Objective: This objective is used to further explore the relationships between local information. By calculating the mutual information between local feature representations of CT and ultrasound images, and using the InfoNCE loss function to optimize the model, it is able to capture the correlations between local details.

[0176] Please see Figure 9 , Figure 9 yes Figure 2 The flowchart further includes step S105. In some embodiments, the process of obtaining the abnormal classification result of the pulmonary artery based on the target fusion image for abnormal classification may include steps S701 to S703:

[0177] Step S701: Perform multiple rounds of convolution processing with different time lengths on the target fused image to obtain the first path feature, and perform multiple rounds of convolution and pooling processing on the target fused image to obtain the second path feature;

[0178] Step S702: The first path features and the second path features are merged to obtain multi-scale features.

[0179] Step S703: Perform classification mapping on the multi-scale features and output the abnormal classification results of the pulmonary artery.

[0180] In the above steps, this embodiment of the present disclosure can extract features from the reconstructed target fusion image using two paths: multi-round asynchronous long convolution and multi-round convolution plus pooling. The final outputs first path features and second path features to capture key symptom features of pulmonary hypertension in the target fusion image from different dimensions. The asynchronous long convolution path focuses on preserving spatial details, while the convolution plus pooling path emphasizes filtering core global features. This dual-path design avoids the shortcomings of a single path, such as losing details or excessive redundancy. Specifically, the first path features, extracted through multi-round asynchronous long convolution, contain spatial detail information of the target fusion image. The second path features, extracted through multi-round convolution plus pooling, contain core global information of the target fusion image.

[0181] Furthermore, the embodiments of this disclosure design a dual-pathway instead of a single-pathway to resolve the contradiction between the multi-scale characteristics of pulmonary hypertension symptoms and the limitations of a single-pathway. The dual-pathway design allows the two features to perform their respective functions. The first path retains local detail anchor points, while the second path provides global trend support. Dual-pathway extraction can effectively cover the multi-scale characteristics of early pulmonary hypertension symptoms.

[0182] Subsequently, in this embodiment, the first path features and the second path features can be merged to obtain multi-scale features. The multi-scale features are a combined feature vector resulting from the merging of the first and second path features. Finally, the multi-scale features are subjected to classification mapping to output the abnormal classification result of the pulmonary artery.

[0183] The classification mapping process is implemented based on a pre-defined fully connected layer and activation function architecture. It progressively maps multi-scale features to a binary classification space, and finally uses a sigmoid activation function to map the output values ​​to the [0,1] interval, obtaining the classification probability. The closer the probability is to 1, the more likely it is pulmonary hypertension; the closer it is to 0, the more likely it is normal. The abnormality classification result is the final output of the classification mapping process, containing two clear conclusions: pulmonary hypertension (abnormal), corresponding to a label of 1; and normal pulmonary artery (no abnormality), corresponding to a label of 0.

[0184] For example, please refer to Figure 10 , Figure 10This is a schematic diagram of the pulmonary hypertension classification model architecture provided in this embodiment. The pulmonary hypertension classification model, as the decoder in this embodiment, adopts a dual-path feature extraction and merging classification structure. Taking the fused image (target image) as input, the upper path sequentially undergoes 3×3 stride 2 convolutions, 3×3 stride 1 convolutions, and 1×1 stride 1 convolutions to adjust the feature size, focusing on preserving subtle local pathological features of the pulmonary artery, resulting in the first path features. The lower path, after normalization by a standard layer, is then processed through 3×3 stride 2 convolutions, pooling layers, and 3×3 stride 2 convolutions to extract the global pathological trend of the pulmonary artery, resulting in the second path features. The first and second path features are merged into multi-scale features via concatenation, then input into a fully connected network for feature mapping. Finally, a classification label is output through a sigmoid activation function as the final anomaly classification result. Based on this, for the fused image, this embodiment constructs a CNN model to classify and diagnose whether the image represents a pulmonary artery abnormality. First, it constructs a model including multiple convolutional layers and pooling layers. The convolutional layers are used to extract image features, capturing local features in the image through convolutional kernels of different sizes (e.g., 3x3, 1x1) and different strides. The pooling layers (e.g., max pooling) are used to reduce the dimensionality of the feature map, reduce computational cost, and improve the model's generalization ability. After feature extraction and fusion, the extracted features are mapped to the classification space through fully connected layers, achieving accurate classification.

[0185] This disclosure also includes experiments and verification of the multimodal pulmonary hypertension image classification model composed of the multimodal image fusion model and the pulmonary hypertension classification model described in the above embodiments:

[0186] This study included 20 patients with pulmonary hypertension and 20 patients with normal pulmonary hypertension. All cases met the 2022 ESC / ERS Guidelines for the Diagnosis and Treatment of Pulmonary Hypertension and also met the following criteria: conforming to the World Health Organization's clinical classification of pulmonary hypertension; complete imaging data (including echocardiography and chest CT), and image quality meeting the requirements of the AI ​​model analysis; relatively stable condition before enrollment, without acute exacerbations or serious complications, such as no acute right heart failure, severe infection, or pulmonary embolism within the past 4 weeks; and age greater than 18 years. Exclusion criteria included: pulmonary venous hypertension caused by left ventricular disease, such as severe mitral stenosis or left heart failure; severe liver or kidney failure; advanced cancer; severe hematologic disorders; pregnancy or lactation; and mental illness. The dataset used in this study included 40 CT images and 40 ultrasound images, respectively from patients clinically diagnosed with pulmonary hypertension and normal subjects, with 20 CT and 20 ultrasound images in each category.

[0187] To ensure data quality and consistency, image standardization and registration were performed as preprocessing steps. First, computer algorithms were used to process the CT images. By learning the mapping relationship between low-resolution and high-resolution images, low-resolution CT images were reconstructed into high-resolution images, thereby improving the spatial resolution of the images. Specifically, CT images were standardized using Hounsfield units (HU), and irrelevant tissues (such as bone and fat) were removed using the professional medical image processing software ITK-SNAP. A linear transformation method was used to convert each pixel value using a linear transformation formula, and the pixel values ​​were scaled to the [0,1] range. For ultrasound images, they were first converted to grayscale, histogram equalization was used to enhance contrast, and normalization was performed. To increase data diversity and improve the model's generalization ability, data augmentation operations were performed on the images. These included: random image rotation ±10 degrees, random horizontal and vertical translation ±10%, random scaling to 90%-110% of the original size, horizontal or vertical flipping with a 50% probability, and adding Gaussian noise with a standard deviation of 0.01. Because CT and ultrasound images operate on different imaging principles, their resolution and viewing angles may differ. A feature-based image registration method (SIFT feature point matching) is used to align CT and ultrasound images to ensure spatial consistency of the multimodal data.

[0188] Finally, the CT and ultrasound images of the selected pulmonary hypertension patients and normal patients after registration are as follows: Figure 11 As shown, Figure 11 This is a schematic diagram of CT and ultrasound images of pulmonary hypertension patients and normal patients provided in the embodiments of this disclosure. From left to right, they are CT images of pulmonary hypertension patients, ultrasound images of pulmonary hypertension patients, CT images of normal subjects, and ultrasound images of normal subjects.

[0189] Next, the preprocessed data was divided into training, validation, and test sets. For example, 70% of the data was used for training, 15% for validation, and 15% for testing, ensuring that each set was representative across different categories (pulmonary hypertension and normal) and modalities (CT and ultrasound), avoiding bias in data distribution. The model's hyperparameters were adjusted, including the learning rate, batch size, number of network layers, and number of neurons. Furthermore, this embodiment used a learning rate decay strategy, with an initial learning rate of 0.001, decaying to 0.1 times the original rate every 10 epochs. The batch size during training was set to 32. A CNN model containing 5 convolutional layers and 3 fully connected layers was designed, with each convolutional layer followed by a batch normalization layer and a ReLU activation function. The final number of neurons in each fully connected layer was set to 512, 256, and 128. All experiments were performed on a local computer, using the Python programming language and the PyTorch deep learning framework to implement the model. Image preprocessing and enhancement operations were performed using the OpenCV and Albumentations libraries.

[0190] Ultimately, the experimental results are as follows: Figure 12 As shown, Figure 12 This is a schematic diagram illustrating the influence of different modules on the change of loss value during the CT and ultrasound multimodal fusion process provided in this embodiment. In the figure, the upper left represents removing the rr module, the upper right represents removing the xx module, the lower left represents removing the cr module, and the lower right represents retaining all modules. The following is a summary of the experimental results: According to Figure 12 Retaining all scales resulted in a rapid decrease in loss from approximately 4.5 to near 0, stabilizing after about 10 epochs, indicating model convergence. Removing the `rr` module resulted in a rapid decrease in loss from approximately 2.5 to near 0, stabilizing after about 10 epochs, indicating faster convergence. Removing the `xx` module also resulted in a rapid decrease in loss from approximately 2.5 to near 0, stabilizing after about 10 epochs, with a similar convergence speed to removing the `rr` module. Removing the `cr` module resulted in a rapid decrease in loss from approximately 4.5 to near 0, stabilizing after about 10 epochs, with a similar convergence speed to retaining all modules. Therefore, removing different modules had little impact on the model's convergence speed and final loss value. All experiments reached a stable state after about 10 epochs. Retaining all modules and removing the `cr` module resulted in higher initial loss values, but similar final convergence results. Removing the `rr` and `xx` modules resulted in lower initial loss values ​​and faster convergence. These results indicate that individual modules have little impact on the final performance of the model, but removing certain modules may accelerate the convergence speed.

[0191] The performance of the multimodal pulmonary hypertension image classification model is as follows: Figure 13 As shown, Figure 13 This is a schematic diagram illustrating the use of a confusion matrix provided in this embodiment to evaluate the performance of a multimodal pulmonary hypertension image classification model. 30% of the samples are used for testing. The horizontal axis represents the predicted results, divided into pulmonary hypertension and normal pulmonary artery types. The vertical axis represents the true results, also divided into normal pulmonary artery and pulmonary hypertension types. Figure 13 From left to right, the graphs show the performance of classification for ultrasound images alone, CT images alone, and images after multimodal fusion. The results are shown in Table 1.

[0192] Table 1, Experimental Results:

[0193]

[0194]

[0195] Table 1 shows that the fused image classification result has the highest accuracy (0.7833), indicating that the fusion method outperforms CT or ultrasound alone in overall classification accuracy. The accuracy of ultrasound alone is 0.7500, slightly higher than CT alone (0.6667). Both CT alone and fused image classification results have a recall of 1.0000, indicating that both methods perform perfectly in identifying all positive examples. The recall of ultrasound alone (0.7143) is relatively low, indicating some missed detections in identifying all positive examples. Ultrasound alone has the highest precision (0.8333), indicating the highest proportion of positive examples among those predicted as positive. The precision of the fused image classification result (0.6833) is lower than ultrasound alone but higher than CT alone (0.6364). The fused image classification result has the highest F1 score (0.8368), indicating a good balance between precision and recall. The F1 score for CT results alone was 0.7778, slightly lower than the fused image classification results, but higher than the 0.7692 for ultrasound results alone. The fused image classification results performed best on most metrics, particularly in accuracy (0.7833 vs 0.7500 for ultrasound).

[0196] The results (0.6667 CT), recall (1.0000), F1 score (0.8368), and AUC (1.0000) indicate that the fusion method can effectively combine the advantages of CT and ultrasound to improve image classification performance. Ultrasound alone showed the best precision (0.8333), but its recall was insufficient (0.7143), resulting in a false negative rate of 28.57% (2 / 7) for pulmonary hypertension cases, indicating the sensitivity limitations of the single-modal model. CT alone performed poorly across all metrics, with an AUC of only 0.7429 and an F1 score of only 0.7778, suggesting that using CT alone for image classification is not the optimal choice for multimodal fusion.

[0197] Based on this, it can be seen that this embodiment essentially proposes a novel multimodal data fusion strategy, combining feature-level fusion with decision-level fusion. At the feature level, deep fusion of features from CT and ultrasound images fully explores the intrinsic connections between the two modalities. At the decision level, combining the decision results of the CNN classifier significantly improves the accuracy and reliability of diagnosis. Experimental results show that the fused image classification results outperform those of single modalities (CT only or ultrasound only) in key metrics such as accuracy (0.7833), recall (1.0000), F1 score (0.8368), and AUC (1.0000), demonstrating the effectiveness of this fusion strategy. For example, in the process of fusing CT and ultrasound images, the model can automatically extract and fuse multi-scale features, compensating for the shortcomings of single-modal data in terms of resolution, imaging principles, etc., thereby significantly improving classification performance.

[0198] Please see Figure 14 This disclosure also provides an anomaly classification device for pulmonary artery images, which can implement the above-described anomaly classification method for pulmonary artery images. The anomaly classification device for pulmonary artery images includes:

[0199] Image acquisition module 1401 is used to acquire target CT images and target ultrasound images of the pulmonary artery;

[0200] The encoding module 1402 is used to encode the target CT image and the target ultrasound image respectively, and extract the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding.

[0201] The fusion module 1403 is used to perform weighted fusion of the first global and local features and the second global and local features through a preset multi-head attention mechanism to obtain multimodal fusion features;

[0202] The decoding module 1404 is used to decode the multimodal fusion features and reconstruct the target fusion image of the pulmonary artery after decoding.

[0203] The classification module 1405 is used to perform anomaly classification based on the target fusion image to obtain the anomaly classification result of the pulmonary artery.

[0204] In summary, the pulmonary artery image anomaly classification device, through the pulmonary artery image anomaly classification method described in the above embodiments, acquires two modalities of images—target CT images and target ultrasound images—and leverages their complementarity to capture potential early symptom features from different dimensions, overcoming the limitations of insufficient information from a single image. Subsequently, the two types of images are encoded separately to extract their global and local features, ensuring that subtle pathological changes are not overlooked while preserving overall structural information. Then, a pre-defined multi-head attention mechanism is used to weightedly fuse the aforementioned multimodal features. This mechanism adaptively focuses on the feature subspace most relevant to pulmonary hypertension, strengthening the expression of key symptom signals and suppressing irrelevant interference, thereby generating more discriminative multimodal fusion features. Next, by decoding and reconstructing the image from these multimodal fusion features, the target fused image is obtained, transforming the fused abstract features into an intuitive and complete image representation. Finally, classification is performed based on the target fused image, enabling the model to more accurately identify the early specific manifestations of pulmonary hypertension, thus overcoming the classification limitations caused by inaccurate symptom identification in related technologies and ultimately improving the accuracy of pulmonary artery image anomaly classification.

[0205] The specific implementation of the pulmonary artery image anomaly classification device is basically the same as the specific embodiment of the pulmonary artery image anomaly classification method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this disclosure, the pulmonary artery image anomaly classification device may also be equipped with other functional modules to implement the pulmonary artery image anomaly classification method in the above embodiments.

[0206] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for classifying abnormalities in pulmonary artery images. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0207] Please see Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0208] The processor 1501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0209] The memory 1502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1502 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1502 and is called and executed by the processor 1501 to execute the abnormal classification method of pulmonary artery images according to the embodiments of this disclosure.

[0210] The input / output interface 1503 is used to implement information input and output;

[0211] The communication interface 1504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0212] Bus 1505 transmits information between various components of the device (e.g., processor 1501, memory 1502, input / output interface 1503, and communication interface 1504);

[0213] The processor 1501, memory 1502, input / output interface 1503 and communication interface 1504 are connected to each other within the device via bus 1505.

[0214] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for classifying abnormalities in pulmonary artery images.

[0215] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of this disclosure. Those skilled in the art will understand that, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this disclosure are also applicable to similar technical problems.

[0216] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0217] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0219] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0220] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0221] In the several embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0222] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0224] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. A method of classifying abnormalities of a pulmonary artery image, characterized by, The method comprises the following steps: obtaining a target CT image and a target ultrasound image of a pulmonary artery; respectively encoding the target CT image and the target ultrasound image, and respectively extracting first global and local features of the target CT image and second global and local features of the target ultrasound image after encoding; performing weighted fusion on the first global and local features and the second global and local features through a preset multi-head attention mechanism to obtain multi-modal fusion features; performing decoding processing on the multi-modal fusion features, and reconstructing a target fusion image of the pulmonary artery after decoding processing; 2. The method of classifying abnormalities of a pulmonary artery image according to claim 1, characterized by, performing abnormal classification based on the target fusion image to obtain an abnormal classification result of the pulmonary artery. The encoding processing of the target CT image and the target ultrasound image respectively, and the extraction of the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding processing, comprises the following steps: respectively encoding the target CT image and the target ultrasound image to obtain first shallow features of the target CT image and second shallow features of the target ultrasound image; respectively performing global feature extraction processing on the first shallow features and the second shallow features to obtain first basic features corresponding to the target CT image and second basic features corresponding to the target ultrasound image; respectively performing local feature extraction processing on the first shallow features and the second shallow features to obtain first detailed features corresponding to the target CT image and second detailed features corresponding to the target ultrasound image; 3. The method of classifying abnormalities of a pulmonary artery image according to claim 2, characterized by, obtaining the first global and local features of the target CT image based on the first basic features and the first detailed features, and obtaining the second global and local features of the target ultrasound image based on the second basic features and the second detailed features. The encoding processing of the target CT image and the target ultrasound image respectively, and the extraction of the first global and local features of the target CT image and the second global and local features of the target ultrasound image after encoding processing, comprises the following steps: respectively inputting the target CT image and the target ultrasound image into a shared encoder, wherein the shared encoder is provided with a multi-head self-attention layer and a feedforward neural network; inputting the target CT image and the target ultrasound image into the multi-head self-attention layer respectively, and outputting first self-attention features of the target CT image and second self-attention features of the target ultrasound image; inputting the first self-attention features and the second self-attention features into the feedforward neural network respectively, and outputting first shallow features of the target CT image and second shallow features of the target ultrasound image.

4. The method of classifying abnormalities of a pulmonary artery image according to claim 2, characterized by, The first global and local features of the target CT image are obtained based on the first basic features and the first detail features, and the second global and local features of the target ultrasound image are obtained based on the second basic features and the second detail features, comprising: obtaining first initial global and local features of the target CT image based on the first basic features and the first detail features, and obtaining second initial global and local features of the target ultrasound image based on the second basic features and the second detail features; The first initial global and local features are input into a preset reversible layer to split the channels of the first initial global and local features into two parts in the reversible layer, the latter part of the channel is incrementally transformed based on the mapping function of the former part of the channel, the former part of the channel is adjusted by product and bias based on the mapping function of the latter part of the channel, and the features of the two parts of the channel are spliced to output the first global and local features of the target CT image after optimization; The second initial global and local features are input into the reversible layer to split the channels of the second initial global and local features into two parts in the reversible layer, the latter part of the channel is incrementally transformed based on the mapping function of the former part of the channel, the former part of the channel is adjusted by product and bias based on the mapping function of the latter part of the channel, and the features of the two parts of the channel are spliced to output the second global and local features of the target ultrasound image after optimization.

5. The method of classifying abnormalities of a pulmonary artery image according to claim 1, wherein, The first global and local features and the second global and local features are weighted and fused by a preset multi-head attention mechanism to obtain multi-modal fusion features, comprising: The first global and local features and the second global and local features are input into a preset multi-head attention fusion module; In the multi-head attention fusion module, the first global and local features are mapped into a query matrix, and the global and local features are mapped into a key matrix and a value matrix, the similarity score of the query matrix and the key matrix is calculated, and the attention weight is obtained after normalization, and the value matrix is weighted based on the attention weight, to splice the outputs of multiple attention heads to obtain multi-modal fusion features.

6. The method of classifying abnormalities of a pulmonary artery image according to claim 1, wherein, The multi-modal fusion features are decoded and processed, and the target fusion image of the pulmonary artery is reconstructed after decoding processing, comprising: The multi-modal fusion features are subjected to feature enhancement processing to obtain multi-modal enhanced features; The multi-modal enhanced features are subjected to dimension reduction processing to obtain multi-modal reduced features; The multi-modal reduced features are subjected to convolution and activation processing to be mapped to the pixel space of the output image for normalization processing, and the target fusion image of the pulmonary artery is output.

7. The method of classifying abnormalities of a pulmonary artery image according to claim 1, wherein, The target fusion image is based on the abnormal classification to obtain the abnormal classification result of the pulmonary artery, comprising: The target fusion image is subjected to multi-round convolution processing with different step lengths to obtain first path features, and the target fusion image is subjected to multi-round convolution and pooling processing to obtain second path features; The first path features and the second path features are subjected to feature merging processing to integrate to obtain multi-scale features; The multi-scale features are classified and mapped to output an abnormal classification result of the pulmonary artery.

8. An abnormality classification device for pulmonary artery images, characterized in that, The method comprises the steps of: An image acquisition module is configured to acquire a target CT image and a target ultrasound image of a pulmonary artery; An encoding module is configured to encode the target CT image and the target ultrasound image respectively, and extract first global and local features of the target CT image and second global and local features of the target ultrasound image after the encoding; A fusion module is configured to perform weighted fusion on the first global and local features and the second global and local features by using a preset multi-head attention mechanism to obtain multi-modal fusion features; A decoding module is configured to decode the multi-modal fusion features and reconstruct a target fusion image of the pulmonary artery after the decoding; A classification module is configured to perform abnormal classification based on the target fusion image to obtain an abnormal classification result of the pulmonary artery.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the abnormal classification method of the pulmonary artery image according to any one of claims 1 to 7 when executing the computer program.

10. A computer readable storage medium, the storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the abnormal classification method of the pulmonary artery image according to any one of claims 1 to 7.

Images