Coronary artery vulnerable plaque identification method and device based on multi-modal large model

Abstract

The invention discloses a coronary artery vulnerable plaque identification method and device based on a multi-modal large model. The method comprises the following steps: acquiring a three-dimensional coronary artery CCTA image to be identified and an SCPR image to be identified; plaque mask information on the blood vessel is obtained; obtaining basic patch data; obtaining fluid mechanics data of each plaque area; obtaining clinical detection basic information; performing feature extraction and feature fusion on the original three-dimensional coronary artery CCTA image, the to-be-recognized SCPR image, the plaque mask information, the basic plaque data and the fluid mechanics data to obtain first fusion data; fusing the first fusion data with the clinical detection basic information to obtain second fusion data; obtaining a vulnerable plaque identification large model; and inputting the second fusion data into the vulnerable plaque identification large model to obtain an identification result. According to the method, the plaque features, the fluid mechanics features and the text features are fused, so that the problems of poor feature expression ability and low recognition performance of a traditional single image are solved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for identifying vulnerable coronary artery plaques based on a multimodal large model and a device for identifying vulnerable coronary artery plaques based on a multimodal large model. Background Art

[0002] Acute coronary syndrome (ACS) has a high mortality rate and accounts for a large proportion of cardiovascular disease deaths. Studies have shown that nearly 70% of ACS are caused by vulnerable plaque rupture. Coronary computed tomography angiography (CCTA) is a recognized non-invasive imaging method for diagnosing coronary arteries. It is also an important method for early screening and definitive diagnosis of coronary heart disease in clinical practice. It can be used to identify coronary artery plaques and assess the degree of vascular stenosis. However, in current clinical applications and diagnostic reports, CCTA assessment is mainly focused on lumen stenosis, and the risk assessment of vulnerable plaques (vulnerable plaques contain four signs, namely positive remodeling, low density, napkin rings and punctate calcification. Plaques containing two or more signs are considered vulnerable plaques) is insufficient. The reason may be that doctors need to make subjective judgments on the characteristics of vulnerable plaques, and the characteristic evaluation is complex, the diagnostic process is time-consuming and labor-intensive, and the professional requirements of doctors are high. There are many international studies on the detection of coronary artery plaques based on CCTA. Maj et al. used RCNN to detect coronary artery plaques. The results obtained can distinguish calcified, non-calcified and mixed plaques according to their spatial distribution. The model has reasonable accuracy in detecting and characterizing coronary artery plaques, but its reliability is not high.

[0003] Therefore, it is desired to have a technical solution to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0004] Application Contents

[0005] The purpose of the present application is to provide a method for identifying vulnerable coronary plaques based on a multimodal large model to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0006] To achieve the above objectives, the present application provides a method for identifying vulnerable coronary artery plaques based on a multimodal large model, and the method for identifying vulnerable coronary artery plaques based on a multimodal large model comprises:

[0007] Acquire the three-dimensional coronary CCTA image to be identified and the SCPR image to be identified;

[0008] Acquiring plaque mask information on the blood vessel according to the SCPR image to be identified;

[0009] Acquire basic plaque data through the plaque mask information;

[0010] Acquiring fluid mechanics data of each plaque area according to the SCPR image;

[0011] Obtain basic information on clinical testing;

[0012] Extracting features from the acquired original three-dimensional coronary CCTA image, the SCPR image to be identified, the plaque mask information, the basic plaque data, and the fluid dynamics data respectively, and fusing the extracted features to obtain first fused data;

[0013] Fusing the first fused data with the basic clinical test information to obtain second fused data;

[0014] Obtain a trained large model for vulnerable plaque identification;

[0015] The second fused data is input into the vulnerable plaque identification large model to obtain an identification result.

[0016] Optionally, the method for identifying vulnerable coronary plaques based on a multimodal large model further comprises:

[0017] The pre-trained large model for vulnerable plaque recognition is fine-tuned.

[0018] Optionally, the fine-tuning training of the pre-trained large vulnerable plaque recognition model includes:

[0019] Obtaining three-dimensional coronary CCTA images, SCPR images, and plaque mask information for training;

[0020] Obtain basic patch data and fluid dynamics data for training;

[0021] Obtain basic information data of clinical tests for training;

[0022] Preprocessing the training three-dimensional coronary CCTA image, SCPR image, and plaque mask information respectively, so as to obtain a first feature vector;

[0023] Performing second preprocessing on the training basic plaque data and fluid mechanics data respectively, so as to obtain a second feature vector;

[0024] Preprocessing the clinical test basic information data for training to obtain a third feature vector;

[0025] The pre-trained large model for identifying vulnerable plaques is fine-tuned using the first feature vector, the second feature vector, and the third feature vector.

[0026] Optionally, the preprocessing of the training three-dimensional coronary CCTA image, SCPR image, and plaque mask information respectively to obtain the first feature vector includes:

[0027] The formats of the three-dimensional coronary CCTA image, the SCPR image, and the plaque mask information are respectively adjusted, thereby obtaining images after the format adjustment;

[0028] Normalizing the images adjusted in various formats to form a normalized image;

[0029] Perform feature extraction on the normalized image to obtain a first feature vector.

[0030] Optionally, performing second preprocessing on the training basic plaque data and fluid mechanics data respectively to obtain a second feature vector comprises:

[0031] Performing data cleaning on basic plaque data and fluid mechanics data for training respectively, thereby obtaining cleaned data;

[0032] Normalizing the cleaned data to obtain normalized data;

[0033] The normalized data is encoded to obtain a second eigenvector.

[0034] Optionally, the preprocessing of the training clinical test basic information data to obtain a third feature vector includes:

[0035] Use the Tokenizer class tool in the Transformers library to convert the basic information of clinical tests used for training into token combinations;

[0036] The encoder is used to extract features from the token combination to obtain the third feature vector.

[0037] Optionally, the fine-tuning training of the pre-trained vulnerable plaque recognition large model by using the first feature vector, the second feature vector and the third feature vector comprises:

[0038] Merging the first eigenvector, the second eigenvector and the third eigenvector to generate a fourth eigenvector;

[0039] Obtaining diagnostic result data;

[0040] Extracting features from the diagnosis result data to obtain a fifth feature vector;

[0041] The pre-trained large model for identifying vulnerable plaques is fine-tuned using the fourth eigenvector and the fifth eigenvector.

[0042] Optionally, the fine-tuning training of the pre-trained vulnerable plaque recognition large model by using the fourth eigenvector and the fifth eigenvector includes:

[0043] Pre-training the pre-trained vulnerable plaque recognition large model for multiple rounds using the fourth eigenvector;

[0044] After completing multiple rounds of preliminary training, freeze the parameters of the fusion module according to the preset step size;

[0045] The fourth eigenvector is compared with the fifth eigenvector to train a large vulnerable plaque model, thereby obtaining a trained large vulnerable plaque model.

[0046] The present application also provides a coronary artery vulnerable plaque identification device based on a multimodal large model, and the coronary artery vulnerable plaque identification device based on a multimodal large model includes:

[0047] An image acquisition module, wherein the image acquisition module is used to acquire a three-dimensional coronary CCTA image to be identified and a SCPR image to be identified;

[0048] A plaque mask recognition module, the plaque mask recognition module is used to obtain plaque mask information on the blood vessel according to the SCPR image to be recognized;

[0049] A basic plaque recognition module, the basic plaque recognition module is used to obtain basic plaque data through the plaque mask information;

[0050] A fluid mechanics data acquisition module, the fluid mechanics data acquisition module is used to acquire fluid mechanics data of each plaque area according to the SCPR image;

[0051] A clinical test basic information acquisition module, wherein the clinical test basic information acquisition module is used to acquire clinical test basic information;

[0052] A first fusion data acquisition module, which is used to extract features from the acquired original three-dimensional coronary CCTA image, the SCPR image to be identified, the plaque mask information, the basic plaque data, and the fluid dynamics data, and fuse the extracted features to acquire first fusion data;

[0053] A second fusion data acquisition module, wherein the second fusion data acquisition module is used to fuse the first fusion data with the clinical detection basic information to obtain second fusion data;

[0054] A vulnerable plaque identification large model acquisition module, wherein the vulnerable plaque identification large model acquisition module is used to acquire a trained vulnerable plaque identification large model;

[0055] The recognition result acquisition module is used to input the second fusion data into the vulnerable plaque recognition large model to obtain the recognition result.

[0056] The coronary vulnerable plaque recognition method based on a multimodal large model of the present application solves the problems of poor expression ability and low recognition performance of traditional single image features by integrating plaque features, fluid dynamics features and text features. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of a method for identifying vulnerable coronary plaques based on a multimodal large model according to an embodiment of the present application.

[0058] Figure 2 It is a schematic diagram of the structure of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0060] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the scope of protection of the present application.

[0061] Figure 1 It is a flowchart of a method for identifying vulnerable coronary plaques based on a multimodal large model according to an embodiment of the present application.

[0062] like Figure 1 The method for identifying vulnerable coronary artery plaques based on a multimodal large model includes:

[0063] Acquire the three-dimensional coronary CCTA image to be identified and the SCPR image to be identified;

[0064] Acquiring plaque mask information on the blood vessel according to the SCPR image to be identified;

[0065] Acquire basic plaque data through the plaque mask information;

[0066] Acquiring fluid mechanics data of each plaque area according to the SCPR image;

[0067] Obtain basic information on clinical testing;

[0068] Extracting features from the acquired original three-dimensional coronary CCTA image, the SCPR image to be identified, the plaque mask information, the basic plaque data, and the fluid dynamics data respectively, and fusing the extracted features to obtain first fused data;

[0069] Fusing the first fused data with the basic clinical test information to obtain second fused data;

[0070] Obtain a trained large model for vulnerable plaque identification;

[0071] The second fused data is input into the vulnerable plaque identification large model to obtain an identification result.

[0072] The coronary vulnerable plaque recognition method based on a multimodal large model of the present application solves the problems of poor expression ability and low recognition performance of traditional single image features by integrating plaque features, fluid dynamics features and text features.

[0073] In this embodiment, clinical information includes basic information of the patient (such as height, weight), physical examination, symptoms and medical history, etc. Biochemical result information mainly includes blood lipid levels (total cholesterol, triglycerides) and blood sugar levels (fasting blood sugar, glycosylated hemoglobin).

[0074] In this embodiment, the SCPR image to be identified is obtained by the following method:

[0075] The coronary artery segmentation model is used to segment and divide the blood vessels, and then the center line of the blood vessel mask is extracted to obtain its SCPR image.

[0076] In this embodiment, obtaining plaque mask information on a blood vessel according to the SCPR image to be identified includes:

[0077] Based on the SCPR image, the plaque mask information on each blood vessel is obtained through the plaque detection model.

[0078] In this embodiment, obtaining basic plaque data through the plaque mask information includes:

[0079] Morphological analysis methods were used to obtain features including but not limited to edge information, length, weight, maximum diameter, density, texture, etc. FFRCT post-processing technology was also used to obtain fluid mechanics features such as velocity, flow rate, and shear force of each plaque area.

[0080] In this embodiment, the above-mentioned various data (original three-dimensional coronary CCTA images, SCPR images to be identified, plaque mask information, basic plaque data and fluid dynamics data) are fused: different data types have very different expressions and dimensional spaces, and they need to adopt their own appropriate preprocessing methods for dimensionality reduction and normalization for fusion and understanding.

[0081] Specifically, in this embodiment, the image data (three-dimensional coronary CCTA image to be identified, SCPR image to be identified, plaque mask information) is processed using the following method:

[0082] The image data is processed in a conventional way, that is, the image is first scaled to a fixed size, such as 256*256*256, and then normalized to standardize the pixel values ​​to the range of [0, 1]. Finally, the image is encoded into a fixed-dimensional feature vector V1 through a convolutional neural network (CNN) or Transform.

[0083] In this embodiment, the structured data (basic plaque data, fluid mechanics data, etc.) are obtained by the following method:

[0084] Structured data refers to data with a predefined format or template, which is generally processed in a standard way, including but not limited to data cleaning (such as missing value processing, consistency checking, outlier detection, etc.), data integration, normalization and data encoding (one-hot encoding or label-encoding), etc., to convert the data into a feature vector V2.

[0085] In this embodiment, text data (i.e., basic information of clinical tests, including biochemical results, clinical information, such as body temperature, etc.) is obtained using the following method:

[0086] The multi-dimensional clinical text data is first converted into a combination of tokens of different lengths through the Tokenizer class tool in the Transformers library, and then the encoder (such as Transformer, BERT) is used to extract features from the tokens to obtain the text feature vector V3.

[0087] In this embodiment, the above-mentioned feature vector V1, feature vector V2 and text feature vector 3 are fused as follows:

[0088] The feature dimensions of image data, structured data and text data conversion vary in size, and generally need to be fused through methods such as PCA, t-SNE or autoencoding. The present invention uses a lightweight convolutional layer to encode the weights of V1, V2 and V3 respectively, and continuously updates and learns during the training process.

[0089] In this embodiment, the method for identifying vulnerable coronary plaques based on a multimodal large model further includes:

[0090] The pre-trained large model for vulnerable plaque recognition is fine-tuned.

[0091] In this embodiment, the vulnerable plaque recognition large model is trained or fine-tuned using the following method:

[0092] The prompt method is used to guide the training of the large model, in which the prompt words are extracted from the diagnosis results. Depending on the final output goals, the extractable prompts include but are not limited to: plaque size, location, vulnerability, degree of danger, and diagnostic opinion.

[0093] Since the diagnosis results are text data, in addition to manually extracting keywords (Tokens), you can also use Tokenizer tools to convert text into Token combinations, and then use the encoder to get the feature vector V5. When training a large vulnerable plaque model, you can use the Fine-Tune method to apply the third and fourth feature vectors to the existing pre-trained multimodal large models (GPT-Vision, PaLM-E, etc.) for fine-tuning.

[0094] In this embodiment, the fine-tuning training of the pre-trained vulnerable plaque recognition large model includes:

[0095] Obtaining three-dimensional coronary CCTA images, SCPR images, and plaque mask information for training;

[0096] Obtain basic patch data and fluid dynamics data for training;

[0097] Obtain basic information data of clinical tests for training;

[0098] Preprocessing the training three-dimensional coronary CCTA image, SCPR image, and plaque mask information respectively, so as to obtain a first feature vector;

[0099] Performing second preprocessing on the training basic plaque data and fluid mechanics data respectively, so as to obtain a second feature vector;

[0100] Preprocessing the clinical test basic information data for training to obtain a third feature vector;

[0101] The pre-trained large model for identifying vulnerable plaques is fine-tuned using the first feature vector, the second feature vector, and the third feature vector.

[0102] In this embodiment, the preprocessing of the training three-dimensional coronary CCTA image, SCPR image, and plaque mask information respectively to obtain the first feature vector includes:

[0103] The formats of the three-dimensional coronary CCTA image, the SCPR image, and the plaque mask information are respectively adjusted, thereby obtaining images after the format adjustment;

[0104] Normalizing the images adjusted in various formats to form a normalized image;

[0105] Perform feature extraction on the normalized image to obtain a first feature vector.

[0106] In this embodiment, the second preprocessing of the training basic plaque data and fluid mechanics data respectively to obtain the second feature vector includes:

[0107] Performing data cleaning on basic plaque data and fluid mechanics data for training respectively, thereby obtaining cleaned data;

[0108] Normalizing the cleaned data to obtain normalized data;

[0109] The normalized data is encoded to obtain a second eigenvector.

[0110] In this embodiment, the preprocessing of the training clinical test basic information data to obtain the third feature vector includes:

[0111] Use the Tokenizer class tool in the Transformers library to convert the basic information of clinical tests used for training into token combinations;

[0112] The encoder is used to extract features from the token combination to obtain the third feature vector.

[0113] In this embodiment, fine-tuning the pre-trained vulnerable plaque recognition large model by using the first feature vector, the second feature vector, and the third feature vector includes:

[0114] Merging the first eigenvector, the second eigenvector and the third eigenvector to generate a fourth eigenvector;

[0115] Obtaining diagnostic result data;

[0116] Extract the features in the diagnosis result data to obtain the fifth feature vector; specifically, the prompt is extracted by text conversion, converting the text into a combination of tokens of different lengths, and then passing through an encoder to obtain the fifth feature vector. There is a certain redundancy in the prompt, and the feature dropout operation is used during training to randomly discard some unimportant prompts.

[0117] The pre-trained large model for identifying vulnerable plaques is fine-tuned using the fourth eigenvector and the fifth eigenvector.

[0118] In this embodiment, fine-tuning the pre-trained vulnerable plaque recognition large model by using the fourth eigenvector and the fifth eigenvector includes:

[0119] Pre-training the pre-trained vulnerable plaque recognition large model for multiple rounds using the fourth eigenvector;

[0120] After completing multiple rounds of preliminary training, freeze the parameters of the fusion module according to the preset step size;

[0121] The fourth eigenvector is compared with the fifth eigenvector to train a large vulnerable plaque model, thereby obtaining a trained large vulnerable plaque model.

[0122] For example, the fourth eigenvector and the fifth eigenvector are trained in comparison: the fourth eigenvector needs to be trained for multiple rounds, and then the parameters of the fusion module are frozen at a certain step size (e.g., step=10), and then the large model is trained in comparison with the fifth eigenvector. The asynchronous training method can make multimodal information fusion faster and large model training easier to converge.

[0123] Whether you are training a large model from scratch or fine-tuning, you need to go through the above training process.

[0124] The present application also provides a coronary artery vulnerable plaque identification device based on a multimodal large model, the coronary artery vulnerable plaque identification device based on a multimodal large model comprises an image acquisition module, a plaque mask recognition module, a basic plaque recognition module, a fluid mechanics data acquisition module, a clinical detection basic information acquisition module, a first fusion data acquisition module, a second fusion data acquisition module, a vulnerable plaque recognition large model acquisition module and a recognition result acquisition module, wherein:

[0125] The image acquisition module is used to acquire the three-dimensional coronary CCTA image to be identified and the SCPR image to be identified;

[0126] The plaque mask recognition module is used to obtain plaque mask information on the blood vessel according to the SCPR image to be recognized;

[0127] The basic plaque identification module is used to obtain basic plaque data through the plaque mask information;

[0128] The fluid mechanics data acquisition module is used to acquire the fluid mechanics data of each plaque area according to the SCPR image;

[0129] The clinical test basic information acquisition module is used to obtain the basic information of clinical tests;

[0130] The first fusion data acquisition module is used to extract features from the original three-dimensional coronary CCTA image, the SCPR image to be identified, the plaque mask information, the basic plaque data and the fluid dynamics data, and fuse the extracted features to obtain the first fusion data;

[0131] The second fusion data acquisition module is used to fuse the first fusion data with the clinical detection basic information to obtain second fusion data;

[0132] The vulnerable plaque identification large model acquisition module is used to obtain a trained vulnerable plaque identification large model;

[0133] The recognition result acquisition module is used to input the second fusion data into the vulnerable plaque recognition large model to obtain the recognition result.

[0134] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.

[0135] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above-mentioned method for identifying vulnerable coronary plaques based on a multimodal large model is implemented.

[0136] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned coronary artery vulnerable plaque identification method based on a multimodal large model.

[0137] Figure 2 It is an exemplary structural diagram of an electronic device capable of implementing the coronary artery vulnerable plaque identification method based on a multimodal large model provided according to an embodiment of the present application.

[0138] like Figure 2As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.

[0139] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1 A method for identifying vulnerable coronary plaques based on a multimodal large model is described.

[0140] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program code; and one or more processors 503 configured to run the executable program code stored in the memory 504 to execute the coronary artery vulnerable plaque identification method based on a multimodal large model in the above-mentioned embodiment.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0145] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0148] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0149] Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0150] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying vulnerable coronary artery plaques based on a multimodal large model, characterized in that: The coronary artery vulnerable plaque identification method based on the multimodal large model includes: Acquire the three-dimensional coronary CCTA image to be identified and the SCPR image to be identified; Acquiring plaque mask information on the blood vessel according to the SCPR image to be identified; Acquire basic plaque data through the plaque mask information; Acquiring fluid mechanics data of each plaque area according to the SCPR image; Obtain basic information on clinical testing; Extracting features from the acquired original three-dimensional coronary CCTA image, the SCPR image to be identified, the plaque mask information, the basic plaque data, and the fluid dynamics data respectively, and fusing the extracted features to obtain first fused data; Fusing the first fused data with the basic clinical test information to obtain second fused data; Obtain a trained large model for vulnerable plaque identification; The second fused data is input into the vulnerable plaque identification large model to obtain an identification result.

2. The method for identifying vulnerable coronary plaques based on a multimodal large model according to claim 1, characterized in that: The method for identifying vulnerable coronary plaques based on a multimodal large model further comprises: The pre-trained large model for vulnerable plaque recognition is fine-tuned.

3. The method for identifying vulnerable coronary artery plaques based on a multimodal large model as claimed in claim 2, characterized in that: The fine-tuning training of the pre-trained vulnerable plaque recognition large model includes: Obtaining three-dimensional coronary CCTA images, SCPR images, and plaque mask information for training; Obtain basic patch data and fluid dynamics data for training; Obtain basic information data of clinical tests for training; Preprocessing the training three-dimensional coronary CCTA image, SCPR image, and plaque mask information respectively, so as to obtain a first feature vector; Performing second preprocessing on the training basic plaque data and fluid mechanics data respectively, so as to obtain a second feature vector; Preprocessing the clinical test basic information data for training to obtain a third feature vector; The pre-trained large model for identifying vulnerable plaques is fine-tuned using the first feature vector, the second feature vector, and the third feature vector.

4. The method for identifying vulnerable coronary plaques based on a multimodal large model according to claim 3, characterized in that: The preprocessing of the training three-dimensional coronary CCTA image, SCPR image, and plaque mask information respectively to obtain the first feature vector includes: The formats of the three-dimensional coronary CCTA image, the SCPR image, and the plaque mask information are respectively adjusted, thereby obtaining images after the format adjustment; Normalizing the images adjusted in various formats to form a normalized image; Perform feature extraction on the normalized image to obtain a first feature vector.

5. The method for identifying vulnerable coronary plaques based on a multimodal large model according to claim 4, characterized in that: The performing second preprocessing on the training basic plaque data and fluid mechanics data respectively to obtain a second feature vector comprises: Performing data cleaning on basic plaque data and fluid mechanics data for training respectively, thereby obtaining cleaned data; Normalizing the cleaned data to obtain normalized data; The normalized data is encoded to obtain a second eigenvector.

6. The method for identifying vulnerable coronary artery plaques based on a multimodal large model according to claim 5, characterized in that: The preprocessing of the clinical test basic information data for training to obtain the third feature vector includes: Use the Tokenizer class tool in the Transformers library to convert the basic clinical test information used for training into a token combination; The encoder is used to extract features from the token combination to obtain the third feature vector.

7. The method for identifying vulnerable coronary plaques based on a multimodal large model according to claim 6, characterized in that: The fine-tuning training of the pre-trained vulnerable plaque recognition large model by using the first feature vector, the second feature vector and the third feature vector comprises: Merging the first eigenvector, the second eigenvector and the third eigenvector to generate a fourth eigenvector; Obtaining diagnostic result data; Extracting features from the diagnosis result data to obtain a fifth feature vector; The pre-trained large model for identifying vulnerable plaques is fine-tuned using the fourth eigenvector and the fifth eigenvector.

8. The method for identifying vulnerable coronary artery plaques based on a multimodal large model according to claim 7, characterized in that: The fine-tuning training of the pre-trained vulnerable plaque recognition large model by using the fourth eigenvector and the fifth eigenvector comprises: Pre-training the pre-trained vulnerable plaque recognition large model for multiple rounds using the fourth eigenvector; After completing multiple rounds of preliminary training, freeze the parameters of the fusion module according to the preset step size; The fourth eigenvector is compared with the fifth eigenvector to train a large vulnerable plaque model, thereby obtaining a trained large vulnerable plaque model.

9. A device for identifying vulnerable coronary plaques based on a multimodal large model, characterized in that: The coronary vulnerable plaque identification device based on the multimodal large model comprises: An image acquisition module, wherein the image acquisition module is used to acquire a three-dimensional coronary CCTA image to be identified and a SCPR image to be identified; A plaque mask recognition module, the plaque mask recognition module is used to obtain plaque mask information on the blood vessel according to the SCPR image to be recognized; A basic plaque recognition module, the basic plaque recognition module is used to obtain basic plaque data through the plaque mask information; A fluid mechanics data acquisition module, the fluid mechanics data acquisition module is used to acquire fluid mechanics data of each plaque area according to the SCPR image; A clinical test basic information acquisition module, wherein the clinical test basic information acquisition module is used to acquire clinical test basic information; A first fusion data acquisition module, which is used to extract features from the acquired original three-dimensional coronary CCTA image, the SCPR image to be identified, the plaque mask information, the basic plaque data, and the fluid dynamics data, and fuse the extracted features to acquire first fusion data; A second fusion data acquisition module, wherein the second fusion data acquisition module is used to fuse the first fusion data with the clinical detection basic information to obtain second fusion data; A vulnerable plaque identification large model acquisition module, wherein the vulnerable plaque identification large model acquisition module is used to acquire a trained vulnerable plaque identification large model; The recognition result acquisition module is used to input the second fusion data into the vulnerable plaque recognition large model to obtain the recognition result.

Images